168 Commits

Author SHA1 Message Date
Paul Spooren
176e1fe191
scripts: dump-target-info print kernel versions
This commits adds the ability to print Kernel versions of all
targets/subtargets. If a testing Kernel is set print that version as
well.

Example output:

    apm821xx/nand 5.10
    apm821xx/sata 5.10
    arc770/generic 5.4
    archs38/generic 5.4
    armvirt/32 5.10
    armvirt/64 5.10
    at91/sam9x 5.10
    at91/sama5 5.10
    ath25/generic 5.4
    ath79/generic 5.4 5.10
    ath79/mikrotik 5.4 5.10
    --- %< ---

This should help to get a quick update on the state of Kernels.

Signed-off-by: Paul Spooren <mail@aparcar.org>
(cherry picked from commit 02de391b086dd2b7a72c2394cfb66cec666a51c1)
2022-01-01 17:26:52 +08:00
Tianling Shen
d3c10b6224
scripts/download.pl: add new mirror
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
(cherry picked from commit 13c56d272d4218a706c986ef37de5be020aeafd6)
2021-11-30 20:44:49 +08:00
Tianling Shen
4c3a0af01c
scripts/download.pl: tweak default mirrors
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-11-29 13:23:45 +08:00
Tianling Shen
923d73b691
scripts/download.pl: reorder self-hosted mirrors
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
(cherry picked from commit d26b4ff4de28af014b55a1040d5dd7fe01c026fb)
2021-11-29 03:31:03 +08:00
Tianling Shen
404f8f2992
scripts/download.pl: add immortalwrt's mirrors
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-11-21 01:07:21 +08:00
Felix Fietkau
cb8dc7b14e toolchain: add a version that can be bumped to force toolchain/target rebuild
This can be used to ensure that the compiled code is up to date, when
something important changes in the toolchain.
A recent example of this is the gcc 11 fix for a code miscompilation issue

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2021-11-19 23:58:57 +08:00
Daniel Golle
aa58b28e47
base-files, metadata: support additional group membership
Some packages may require additional group membership for the system
user added by that package. Allow defining additional groups as third
member of the ':'-separated tuple, allowing to specify multiple
','-separated groups with optional GID.

Example:
USERID:=foouser=1000:foogroup=1000:addg1=1001,addg2=1002,addg3

Signed-off-by: Daniel Golle <daniel@makrotopia.org>
(cherry picked from commit b2aca61360c33ad3161f8ef35a1c28ebb2507600)
2021-11-10 15:42:57 +08:00
Josh Soref
e72185a4a1 scripts: fix various typos
This only affects typos in comments or user-facing output.

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
[only picks changes to scripts, drop "commandline" replacement,
 fix case for "arbitrary", improve commit message]
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2021-11-02 23:49:21 +08:00
Jan-Niklas Burfeind
cffc1cd77b scripts: eva_ramboot.py: remove unused import
concludes:
commit e7bc8984d9ca ("scripts: make eva_ramboot.py offset configurable")

Signed-off-by: Jan-Niklas Burfeind <git@aiyionpri.me>
2021-11-02 23:46:05 +08:00
Petr Štetiar
dc89ad3dfe
scripts/gen_image_generic.sh: fix more shellcheck warnings
Fixes following shellcheck warnings:

 In scripts/gen_image_generic.sh line 20:
 cyl=$(( (KERNELSIZE + ROOTFSSIZE) * 1024 * 1024 / (head * sect * 512)))
 ^-^ SC2034: cyl appears unused. Verify use (or export if used externally).
 --
 In scripts/gen_image_generic.sh line 34:
     [ -n "$PADDING" ] && dd if=/dev/zero of="$OUTPUT" bs=512 seek="$(($ROOTFSOFFSET + $ROOTFSSIZE))" conv=notrunc count="$sect"
                                                                       ^-----------^ SC2004: $/${} is unnecessary on arithmetic variables.
 --
 In scripts/gen_image_generic.sh line 35:
     mkfs.fat -n kernel -C "$OUTPUT.kernel" -S 512 "$(($KERNELSIZE / 1024))"
                                                       ^---------^ SC2004: $/${} is unnecessary on arithmetic variables.

Signed-off-by: Petr Štetiar <ynezz@true.cz>
(cherry picked from commit 92d7cedcad73ff9845ea3f635ea82bd925078c1f)
2021-10-14 21:25:07 +08:00
Paul Spooren
7218521845 scripts: package-metadata add pkgmanifestjson call
The new `pkgmanifestjson` call prints all package manifest of a feed in
JSON format. This function can be used to print an overview of packages
information used for downstream tooling.

The script is entirely based on Petrs work on dependency visualisation.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2021-10-12 23:56:16 +08:00
Paul Spooren
736c627160 scripts: store maintainer in package metadata
The maintainer could be usable for downstream tooling, so start storing
it in the metadata.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2021-10-12 23:55:51 +08:00
Ansuel Smith
2969a245ec scripts: handle gcc and linux in dl_cleanup script
Handle gcc and linux with a special regex that set their progname with
their major version. This way every minor version can be cleared. The
build cleanup logic can be tweaked later to clean the entire toolchain
and target dir with a different gcc version.

Signed-off-by: Ansuel Smith <ansuelsmth@gmail.com>
[reformat commit message]
Signed-off-by: Paul Spooren <mail@aparcar.org>
2021-10-12 23:51:27 +08:00
Ansuel Smith
7b630b974e scripts: format dl_cleanup to black format python style
Forma dl_cleanup python script to black style.

Signed-off-by: Ansuel Smith <ansuelsmth@gmail.com>
2021-10-12 23:51:14 +08:00
Ansuel Smith
6477cae7a1 scripts: add missing regex for dl_cleanup script
Regex xxx-YYYY-MM-DD-GIT_SHASUM was missing. Add the new regex to improve
and better find outdated package. This also fix a bug where some bug were
incorrectly detected as packagename-yyyy-mm-dd instead of packagename due
to them be parsed by the wrong parser

Example:

    openwrt-keyring-2021-02-20-49283916.tar.xz

Signed-off-by: Ansuel Smith <ansuelsmth@gmail.com>
[added example in commit message]
Signed-off-by: Paul Spooren <mail@aparcar.org>
2021-10-12 23:50:56 +08:00
Paul Spooren
e9cb0a7753
build: store artifacts in JSON
Multiple profiles create artifacts, these should be stored in the JSON
file as well, allowing downstream tooling to show those files, too.

Artifacts don't have specific filesystems so only the fields `name`,
`type` and `sha256` are available.

Rename env variable names from IMAGE_ to FILE_ prefixes to reflect that
images, kernels and artifacts are added with the same command.

Signed-off-by: Paul Spooren <mail@aparcar.org>
(cherry picked from commit 5ef4608c02ef6db9698c8d10011b9c14a8ca073b)
2021-09-17 04:49:04 +08:00
Adrian Schmutzler
7984a95946
treewide: rename IMAGE_PREFIX/IMAGE_NAME to DEVICE_IMG_*
We so far had two variables IMG_PREFIX and IMAGE_PREFIX with
different content. Since these names are obviously quite
confusing, this patch renames the latter to DEVICE_IMG_PREFIX,
as it's a device-dependent variable, while IMG_PREFIX is only
(sub)target-dependent.

For consistency, also rename IMAGE_NAME to DEVICE_IMG_NAME, as
that's a device-dependent variable as well.

Cc: Paul Spooren <mail@aparcar.org>
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
(cherry picked from commit ef2cb8572b48f1e1964b4d1d014d16cb721b5175)
2021-09-17 04:48:26 +08:00
SuLingGG
e45c211c5a
scripts/download.pl: update openwrt.cc mirror url
[updated url]
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
(cherry picked from commit 009c17217107834cdf5ccc336e791ef0b33eada1)
2021-08-30 23:07:46 +08:00
Tianling Shen
819db9dc2b
scripts/download.pl: re-add aliyun mirrors
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-07-15 22:40:00 +08:00
Ansuel Smith
8699f787e0 scripts: check if dl directory exist in dl_cleanup script
Check if the provided dl directory exist and return on error.

Signed-off-by: Ansuel Smith <ansuelsmth@gmail.com>
2021-07-06 23:06:50 +08:00
Paul Spooren
a481251f47
build,json: fix generation with empty profiles
If the image generation doesn't add any profiles to the output the
*profile merge* will fail. To avoid that set an empty profile as
fallback.

Signed-off-by: Paul Spooren <mail@aparcar.org>
(cherry picked from commit fd0d9909bf50f114d5e7f7cedf53e542de878a2c)
2021-06-21 20:35:32 +08:00
Tianling Shen
a3325e5051
download.pl: add some mirrors disabled by default
To avoid abuse, these mirrors will not be used until you enable them
manually.

Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-06-19 18:30:22 +08:00
AmadeusGhost
4b33d667e8 download.pl: refresh mirrors list
Since there are so many useless actions, just to catch up with
the 'new update' ...
2021-06-19 18:08:10 +08:00
AmadeusGhost
76b990208f
download.pl: add more mirrors for SourceForge
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-06-19 00:46:41 +08:00
Moritz Warning
2cf729818b
build: preserve profiles.json between builds
Keep other profiles.json content if the data belongs to the current
build version.

Also useful for the ImageBuilder, which builds for a single model each
time. Without this commit the profiles.json would only contain the
latest build profile information.

Signed-off-by: Moritz Warning <moritzwarning@web.de>
[improve commit message]
Signed-off-by: Paul Spooren <mail@aparcar.org>
(cherry picked from commit a463b96241fbc2d142982387eaed9989e201ac7a)
2021-06-17 20:53:42 +08:00
李国
9eddf65670
x86: generate EFI platform bootable images
Add EFI platform bootable images for x86 platforms. These images can
also boot from legacy BIOS platform.

EFI System Partition need to be fat12/fat16/fat32 (not need to load
filesystem drivers), so the first partition of EFI images are not ext4
filesystem any more.

GPT partition table has an alternate partition table, we did not
generate it. This may cause problems when use these images as qemu disk
(kernel can not find rootfs), we pad enough sectors will be ok.

Signed-off-by: 李国 <uxgood.org@gmail.com>
[part_magic_* refactoring, removed genisoimage checks]
Signed-off-by: Petr Štetiar <ynezz@true.cz>
(cherry picked from commit a6b7c3e672764858fd294998406ae791f5964b4a)
2021-06-17 20:13:41 +08:00
Karel Kočí
f3e47345b1 scripts/feeds: generate index after all feeds are updated
This separates index update from feed update. The result is that all
requested feeds are first updated and only then indexed.

The reason for this change is to prevent errors being reported and
potentially invalid index being generated thanks to cross feeds
dependency.
The feeds script pulls in default all feeds as they come and on install
prefers packages from first feeds (unless special feed is requested).
Thus order of feeds in some way specifies preferences. This is handy for
downstream distributions as they can simply override any package from
upstream feeds by placing their feed before them. This removes need to
patch or fork upstream feeds.
The problem is that such feed most likely depends in some way also on
subsequent feeds. The most likely feeds are 'packages' or 'luci'. The
example would be Python package that needs 'python.mk' from 'packages'
feed. Ordering custom feed after dependent feeds is sometimes just not
possible because of preference requirement described before.
The solution is to just first pull all feeds and generate indexes only
after that. In the end this ensures that index is generated correctly at
first try without any error.

In terms of code this removes 'perform_update' argument from
'update_feed' as with index update removal the update is the only action
performed in that subroutine. Thus this moves condition to 'update'
subroutine.

Signed-off-by: Karel Kočí <karel.koci@nic.cz>
2021-06-08 23:16:09 +08:00
Paul Spooren
2c11e3e346 scripts: config.guess: update to 2021-05-24
This script hasn't seen an update in multiple years, update it to the
latest version provided upstream. Both `config.guess` and `config.sub`
are copied from upstream[1] and not modified.

The full changelog is available within the upstream repository[1].

[1]: https://git.savannah.gnu.org/git/config.git

Signed-off-by: Paul Spooren <mail@aparcar.org>
2021-05-29 23:21:13 +08:00
Tianling Shen
f6ef5a895c
scripts/download: update mirror url
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-05-26 22:17:21 +08:00
Leonardo Mörlein
2a821d862c
build: introduce $(MKHASH)
Before this commit, it was assumed that mkhash is in the PATH. While
this was fine for the normal build workflow, this led to some issues if

    make TOPDIR="$(pwd)" -C "$pkgdir" compile

was called manually. In most of the cases, I just saw warnings like this:

    make: Entering directory '/home/.../package/gluon-status-page'
    bash: line 1: mkhash: command not found
    bash: line 1: mkhash: command not found
    bash: line 1: mkhash: command not found
    bash: line 1: mkhash: command not found
    bash: line 1: mkhash: command not found
    bash: line 1: mkhash: command not found
    bash: line 1: mkhash: command not found
    bash: line 1: mkhash: command not found
    [...]

While these were only warnings and the package still compiled sucessfully,
I also observed that some package even fail to build because of this.

After applying this commit, the variable $(MKHASH) is introduced. This
variable points to $(STAGING_DIR_HOST)/bin/mkhash, which is always the
correct path.

Signed-off-by: Leonardo Mörlein <me@irrelefant.net>
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-05-13 22:49:42 +08:00
Tianling Shen
1b76d7006b
scripts/download.pl: only call our own mirror when failed to download
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-04-19 01:36:37 +08:00
Tianling Shen
29b53e375a
scripts/download.pl: add aliyun(cn) mirror
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-04-19 00:15:44 +08:00
Tianling Shen
9961791b9e
scripts/download.pl: remove unavailable mirror
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-04-17 00:12:02 +08:00
Tianling Shen
4f53a26f06
scripts/download.pl: update mirrors
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-04-13 21:33:25 +08:00
Tianling Shen
3364f9a63d
scripts/download.pl: re-order mirrors
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-03-26 22:16:38 +08:00
Paul Spooren
5ec2e9b8c7 build,json: 3rd fixup of default_packages
This became a bit of a tragedy, caused by a corner cases which wasn't
put into account during testing. DEFAULT_PACKAGES are defined in
target/linux/<target>/Makefile but also in
target/linux/<target>/<subtarget>/target.mk.

The latter was no longer imported when using DUMP=1, however not using
DUMP=1 while running the Makefile in target/linux/<target>/ caused duplicate
packages in the list.

As a solution, which should have been used from day 0, `make` runs in
target/linux/ without DUMP=1, resulting in no duplicate packages and all
inclusions from include/target.mk, linux/target/<target>/{Makefile,
<subtarget>/target.mk}

While at it, sort the list of default packages.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2021-03-26 20:00:19 +08:00
Paul Spooren
d28cf096e3 build,json: fixup fixup of arch_packages
The commit "1bf2b3fe90 build,json: fixup missing arch_packages" fixes
the missing package architecture locally but runs $(TOPDIR)/Makefile
rather than a target specific one. While this works on local builds just
fine, it causes the buildbots to add garbage to the `arch_packages`
variable:

    cd \"/builder/shared-workdir/build\"; git log --format=%h -1
    toolchain > /builder/shared-workdir/build/tmp/.ver_check\ncmp -s
    /builder/shared-workdir/build/tmp/.ver_check
    /builder/shared-workdir/build/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/stamp/.ver_check
    || { \\\n\trm -rf
    /builder/shared-workdir/build/build_dir/target-x86_64_musl
    /builder/shared-workdir/build/staging_dir/target-x86_64_musl
    /builder/shared-workdir/build/staging_dir/toolchain-x86_64_gcc-8.4.0_musl
    /builder/shared-workdir/build/build_dir/toolchain-x86_64_gcc-8.4.0_musl;
    \\\n\tmkdir -p
    /builder/shared-workdir/build/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/stamp;
    \\\n\tmv /builder/shared-workdir/build/tmp/.ver_check
    /builder/shared-workdir/build/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/stamp/.ver_check;
    \\\n}\nx86_64

Only the last line contains the desired string.

Future investigation should check why the build system prints this to
stdout rather than stderr.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2021-03-25 23:38:15 +08:00
Paul Spooren
65ca8a8b72 build,json: fixup missing arch_packages
Fix 7f4c2b1 "build,json: fix duplicates in default_packages" which
removed duplicate default packages but also removed the package
architecture from the profiles.json.

If DUMP=1 is set, the `ARCH_PACKAGES` is no longer exported and
therefore empty. Fix this by running make twice, once with DUMP=1 and
once without.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2021-03-25 23:36:48 +08:00
Tianling Shen
903bd12b2b
scripts/download.pl: add self-hosted mirror
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-03-23 23:32:35 +08:00
Tianling Shen
49e3204f02
Revert "treewide: add mirrors for GitHub Codeload"
This reverts commit 9796619efca52ecfb76a41dd5d15606380f55914.

Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-03-23 16:58:11 +08:00
Tianling Shen
66a13ba4b4
Revert "treewide: add mirrors for GitHub Archives"
This reverts commit 19980fd5b99b03aa0e3874127d8ed396db5f7545.

Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-03-23 16:58:11 +08:00
Tianling Shen
a53b7d6d62
scripts/download.pl: add download mirrors hosted by SuLingGG
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-03-22 23:21:13 +08:00
Tianling Shen
9f898e3336
download.pl: fix path to mirror 182.140.223.146
Closes: #327

Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-03-22 17:10:15 +08:00
Paul Spooren
e2ce5345c8 build,json: fix duplicates in default_packages
Calling without the DUMP=1 argument causes the target specific Makefile
to be "included" again which adds the target specific packages twice,
once on the actual run and once included from `include/target.mk`.

This led to duplicate package entries, causing confusion in downstream
projects using the generated JSON files.

While at it, apply `black` style to Python script.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2021-03-21 23:36:45 +08:00
Paul Spooren
c405955a41 scripts,ipkg-build: use realpath for pkg_dir
This allows manual execution of the ipkg-build script even with
releative path.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2021-03-16 11:13:58 +08:00
Paul Spooren
df8b658cb5 scripts,ipkg-build: apply shellcheck
This commit cleans the `ipkg-build` script via changes suggested by
shellcheck. These are mostly word splitting issues.

Remove the definition of GZIP, this adds three "lookups" of the `gzip`
binary but the rest of the build system doesn't seem to use such
improvements neither.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2021-03-16 11:13:15 +08:00
Tianling Shen
19980fd5b9
treewide: add mirrors for GitHub Archives
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-03-15 15:01:44 +08:00
Tianling Shen
9796619efc
treewide: add mirrors for GitHub Codeload
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-03-15 14:17:10 +08:00
Tianling Shen
93dfa53568
scripts/download.pl: add global mirror for mainland China users
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-03-15 11:19:22 +08:00
Tianling Shen
99a9f78ee3
treewide: adjust tencent mirror
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-03-11 14:57:01 +08:00