pkgname is the name of the package, with no restrictions on allowed characters.pkgver in is the combination of the pkgname, a version and a revision in the form of pgkname-version_revision.version is the release version of a package.
- (dash).. (dot), alpha, beta, pre, rc) influence the comparison between versions.revision is a number that is added to the version to be able to update/change a package without a version increase.pkgpattern is the combination of a pkgname, a operator and a version.pkgver ::= name "-" version "_" revision
name ::= [a-zA-Z0-9-]*
version ::= [^-_]*
revision ::= [0-9]+
pkgpattern := name operator .+
operator ::= ( ">=" ">" / "<=" / "<" / "==" / "!=" )
The version separator - is common in package names and makes distinguishing pkgver and plain package names hard.
→ Using @ (at) as separator would have been a better choice, because it is a lot more uncommon in package names.
The revision separator _ has the same issue as the version separator.
Dependency solving is one of the most important and hard problems about package management.
When updating a package, files can move from one to another package. If a file moves from package A to package B, the update of package A will delete the file and the update of package B will recreate it.
This makes the order of the installation important, if A is updated before B
14:26:25 duncaen | its a bug
14:26:56 duncaen | but also an issue
14:27:10 duncaen | hostdir has priority over the remote repos
14:27:39 duncaen | xbps-src will install direct dependencies as xbps-install -S gnutls-3.6.12_1
14:27:51 duncaen | this includes the version and will install the repository package
14:28:13 duncaen | but if something depends on gnutls it will choose the one from the first repository, hwich is hostdir
14:28:53 duncaen | then there is a bug in xbps where it adds a second version of the package which should be fixed in HEAD
14:29:18 quinq | ok, so it's all taken care of ^^
14:29:29 duncaen | there is a fundamental flaw
14:29:44 duncaen | there is the bestmatching=yes configuration option which should choose the greatest version
14:29:58 duncaen | we should probably use that in xbps-src to prioritize things correctly
14:30:37 duncaen | the fix in xbps should not allow two version of a package
14:30:57 duncaen | but it also shouldn't recover from this because this would be the same thing as bestmatching=yes
14:31:32 duncaen | but then again, bestmatching=yes is the sane default and I don't really know why this is even an option
14:31:46 quinq | Yeah, but it's the default option
14:32:18 quinq | oh
14:32:18 quinq | # Enable package best matching (disabled by default). If enabled a package with
14:32:21 quinq | #bestmatching=true
14:32:22 quinq | No it isn't
14:32:40 duncaen | yes
14:32:56 duncaen | not sure if there is anything depending on that
14:33:12 duncaen | we have repolock for cases we want to prioritize one repository
14:33:27 duncaen | s/prioritize/lock a package to a custom repo
14:34:00 duncaen | but I guess that solves all issues
14:34:04 quinq | In any case, when changing a default bit, you can almost be sure somebody decided to depend on it at some point, wherease it's the correct way or not ;)
14:34:26 duncaen | at the moment you can roll your own repository with a bunch of packages downgraded and make sure it comes before the upstream repo
14:34:36 quinq | ok, this solves the issue, thanks
14:34:38 duncaen | with bestmatching=yes by default this wouldn't work anymore
14:34:50 quinq | But you can hold them
14:35:04 duncaen | hold/repolock, but you can't install them
14:35:12 quinq | right
14:35:23 duncaen | same issue but the other way around I guess lol
14:35:38 duncaen | you can specify the version on the command line but that doesn't work for dependencies
14:37:56 duncaen | I think there should be a way in xbps-src to warn or error out in cases like this
14:38:03 duncaen | and a easy way to clean the repository
16:01:27 duncaen | virtual packages as example are weird and under specified
16:01:52 duncaen | they were never designed with the idea how allowing to install multiple packages that provide the same virtual package
16:02:16 duncaen | since xbps-alternatives was added, its easy to have multiple virtual package provides installed
16:02:27 duncaen | as example gawk and nawk
16:03:15 duncaen | with how virtual packages work, in xbps the alphabetically last installed real package for a virtual provider takes precedence
16:03:39 duncaen | so if you have gawk and nawk installed, and you want to uninstall nawk it is now not possible without force
16:04:19 duncaen | because reverse dependencies depending on the awk virtual package only know about the alphabetically last provider, nawk and xbps thinks removing nawk will break the packages with the awk
| dependenct
16:04:22 duncaen | *dependency
16:05:11 duncaen | in general a lot is based on alphabetically first alphabetically last or installed first installed last
16:05:18 duncaen | and there is no way to change this behaviour
16:05:50 duncaen | like with xbps-alternatives, the first package in the transaction will provide the alternative
16:05:59 leah2 | it's not well-designed but fairly hackable and we fixes most annoying things over time :p
16:06:22 leah2 | duncaen: the alternative [heh] is a debian-complexity alternatives system
16:06:24 duncaen | the order of dependencies in base-system is critical to make it work
16:06:33 leah2 | which uses random integers people assign to packages...
16:07:07 duncaen | gawk has to come early in base-system dependencies before any package depends on the awk virtual package, otherwise xbps would install chroot-awk, because its alphabetically first and then
| conflict with gawk
16:07:24 duncaen | so virtual package if not installed then alphabetically first, if installed alphabetically last
16:07:27 duncaen | this is madness xD
14:51:26 q66 | duncaen: i don't think it's possible to handle noarch really well because regarldess of what you do you'll have noarch packages that are tied to non-noarch packages somehow and those
| non-noarch packages may be platform specific
14:51:37 q66 | then managing what belongs to which index becomes non-obvious/a pain in the ass
14:51:59 q66 | i don't think there's much point to it either way considering space is the cheapest thing
14:52:29 duncaen | yea for sub packages, noarch is bad
14:52:30 q66 | and as for bandwidth, people will fetch the same amount of stuff regardless of whether it's one file or several files
14:52:37 q66 | not subpackages
14:52:50 q66 | i mean noarch packages (e.g. a python program) that depend on native libraries (e.g. through ffi, or indirectly)
14:53:03 duncaen | hm right, they shouldn't be noarch
14:53:17 q66 | the thing is, just about anything depends on something native along the awy
14:53:19 q66 | *way
14:53:38 q66 | and the few things that remain make up relatively little
14:53:43 duncaen | was thinking about some of the -data packages, but yea they also depend on non noarch
14:53:54 q66 | yeah -data depend directly on arch specific
14:54:02 duncaen | even meta packages would be bad noarch packages
14:54:25 q66 | also, noarch + build options don't go together either
14:54:45 q66 | because if you assume that a build option might differ depending on the environment it's run in
14:54:50 duncaen | probably the easiest and best way to handle it is to drop noarch altogether
14:54:52 q66 | and you also make the assumption that noarch packages should be shared
14:54:55 q66 | that also conflicts
14:55:00 q66 | a notable case for this was qutebrowser
14:55:06 q66 | a python program, originally noarch package
14:55:10 q66 | it has webkit and webengine backends
14:55:23 q66 | webengine is preferred on x86/aarch64 but everywhere else you need to use webkit
14:55:36 q66 | yeah that's what i'm thinking
13:04:56 duncaen | staging kinda works already
13:05:09 Piraty | better than before staging
13:05:09 duncaen | the problem comes when there are multiple independent repositories
13:05:30 duncaen | xbps-rindex and the staging stuff only ever works on one repository
13:06:28 duncaen | while the repo containing libffi is most likely in staging while revdeps are being build, other repositories like nonfree cant stay in staging because they are only aware of itself and ignore
| to stage missing shlibs from other repositories
13:08:14 duncaen | so the main repo is staged and things depends on glib-2.62.5_1 while another repositorie is not staged and depend on glib-2.62.5_2, which isn't available in the main repository
13:24:55 visitorius | can not this be coordinated among repos somehow?
13:26:52 duncaen | sure, but it isn't
13:27:16 duncaen | and it would intorduce new issues
13:27:35 duncaen | binary packages in the nonfree repo at the moment don't lock our main repository into staging
13:28:01 duncaen | a side effect that breaks those binary packages but at least not our main repository
13:28:53 duncaen | but adding something that acts on multiple repos doesn't really work well with the current tooling
13:29:08 duncaen | xbps-rindex works on one repository and changing that doesn't seem right
13:30:04 duncaen | the optimal solution would be to remove staging code form xbps-rindex, let xbps-src still use xbps-rindex and _always_ stage
13:30:53 duncaen | then have a new tool that acts on multiple repositories and possible for multiple architectures that then goes through the staged packages and selectively puts packages that are complete into
| the "published" repository
13:32:08 duncaen | also arch linux besides not having the shlib stuff, they are also not using a CI like we do afaik
13:32:29 duncaen | and they have a testing repository
13:36:17 visitorius | such would be probably a quite big change
13:37:29 visitorius | but do i understand it properly, that would i have not been using nonfree repo (i use it mainly due to nvidia, i think) i wouldn't have this problem right now?
13:45:54 duncaen | not really, its not only nonfree any case where there are multiple repositories
13:46:14 duncaen | this includes -debug and -multilib
13:46:49 visitorius | thank you
13:47:10 duncaen | and the staging stuff made this already a lot better
13:47:30 duncaen | before that this things happened for each build
13:47:37 visitorius | yes it seems i ma hitting it much less often, i think about it
13:47:52 visitorius | ... when i think ...
14:02:37 visitorius | thanks guys
14:13:11 st3r4g[m] | duncaen: sounds cool
14:13:19 st3r4g[m] | why also for multiple arches tho?
14:13:42 duncaen | mainly noarch
14:13:56 st3r4g[m] | oh right
14:14:06 duncaen | alternative would be to drop noarch
14:14:57 st3r4g[m] | it's kind of a singular case
14:15:06 duncaen | changing xbps-rindex to always stage wouldn't even make a lot of differencesd
14:15:20 duncaen | you can still use -stagedata from you local repository
14:15:43 duncaen | a new step for publishing packages wouldn't make any difference because you already have to sign packages for remote repositories in a separate step
2020-01-29 16:00:46 bapt xbps-query -Ro, what does it uses as a backend, a db of files that is fetched?
2020-01-29 16:00:52 bapt if yes what size?
2020-01-29 16:02:09 shahab https://voidlinux.org/faq/#searching-a-file
2020-01-29 16:02:37 bapt thanks
2020-01-29 16:02:54 shahab bapt, that does not answer your question.
2020-01-29 16:03:20 bapt nope it does not ;)
2020-01-29 16:03:22 duncaen xbps-query -Ro fetches the each package file, which is slow
2020-01-29 16:03:41 duncaen it fetches the beginning of the package archives, files.plist is one of the first files
2020-01-29 16:03:49 shahab yeah, it says on the last paragraph here: https://docs.voidlinux.org/xbps/packages/files.html
2020-01-29 16:04:13 duncaen we don't have a file database in xbps, xlocate is used as a workaround
2020-01-29 16:04:59 bapt ok
2020-01-29 16:05:02 duncaen I was thinking about a file database, but I think because you asked about the size you know the issue
2020-01-29 16:05:13 bapt yes
2020-01-29 16:05:42 bapt but I am in my case for freebsd close to a text format which is "acceptable" to download
2020-01-29 16:05:50 bapt less than 10M xzed compress
2020-01-29 16:06:12 bapt for 31616 packages
2020-01-29 16:06:48 duncaen would be interesting to see if a tree would save a lot of space
2020-01-29 16:06:58 bapt it does ;)
2020-01-29 16:07:06 bapt that is what I use
2020-01-29 16:07:25 bapt kind of a mini mtree format
2020-01-29 16:07:25 duncaen ah nice, I have to look more at pkgng
2020-01-29 16:07:33 bapt it is not in there yet
2020-01-29 16:08:01 bapt only locally as I have only been experimenting with it, but not enough to be sure this is a viable solution yet
2020-01-29 16:08:23 bapt and I am now exploring what others are doing to see if there nothing obvious I could also use :D
2020-01-29 16:08:31 bapt before going my NIH way
2020-01-29 16:08:54 duncaen yes the NIH is always strong with package managers, but I don't think I've seen any nice solution like the tree
2020-01-29 16:10:02 bapt the file db is big for debian and fedora
2020-01-29 16:10:09 bapt no tree there
2020-01-29 16:11:32 duncaen how large are your repository databases compared to the file database?
2020-01-29 16:12:23 bapt locally 50M for the full repository (freebsd current)
2020-01-29 16:12:42 bapt the file database is around 80M in tree mode so far
2020-01-29 16:12:55 bapt (the full repository so far is sqlite backed)
2020-01-29 16:13:33 bapt I have experimented using netbsd's cdb for the full repository, I can strip down the size to 25M,
2020-01-29 16:13:46 bapt but too much work for now to finish that experimentation
2020-01-29 16:13:52 duncaen we only have like ~7k packages so our database is a lot smaller
2020-01-29 16:14:30 bapt the full db to download is (xzed compress) 6 MiB
2020-01-29 16:14:49 duncaen 1.9M compressed and 14.4M uncompressed because its a big plist file
2020-01-29 16:15:13 bapt this is a text file (one inline yaml/json per line) which is injected to a sqlite db
2020-01-29 16:15:59 bapt I use a db, because the look up where very painfully slow with flat file db model when having more than 20k packages
2020-01-29 16:16:01 duncaen thats not bad, iirc debian/fedora repository syncs are a lot larger
2020-01-29 16:16:03 bapt before it was ok
2020-01-29 16:16:31 duncaen yea makes sense with that many packages
2020-01-29 16:16:57 bapt but given how I use the remote db a constant database would have more sense than a sqlite in the end
2020-01-29 16:17:10 bapt at each sync I just recreate the db each time
2020-01-29 16:17:22 bapt so if one day I have motivation I will finish the cdb backend ;)
2020-01-29 16:17:29 duncaen cdb is the djb constant db?
2020-01-29 16:17:33 bapt nope
2020-01-29 16:17:38 bapt netbsd's cdb
2020-01-29 16:17:42 bapt cdbr cdbw
2020-01-29 16:18:12 bapt https://github.com/rmind/libcdb
2020-01-29 16:18:54 bapt they use it in their libtinfo, there services(5) database and they npf firewall
2020-01-29 16:19:11 bapt iirc they patched their postfix to also support it as a database format but unsure
2020-01-29 16:21:29 anddam shahab: the faq is a bit confusing since for 'xlocate -S' shows the git cloning that I do not see atm
2020-01-29 16:21:43 anddam but other than that seems a brilliant solution, thanks
2020-01-29 16:22:32 anddam duncaen: troglobit/mdsnd is not packaged for void though, right?
2020-01-29 16:23:16 duncaen anddam: right, but thats more for exposing services, not like you want just .local resolution
2020-01-29 16:25:03 duncaen bapt: how would a cdb package database look like, would it be one big database with pkgname-key for each bit of information?
2020-01-29 16:25:57 bapt I have taken a weird approach to fall under 25MB
2020-01-29 16:26:35 bapt I have 2 types of data for a package: strings and integer
2020-01-29 16:26:52 bapt I dedup all the string first
2020-01-29 16:27:31 bapt meaning I only push data and keep an index of where the data is located in the cdb (but I do not push any key pointing at the data)
2020-01-29 16:27:44 duncaen anddam: xbps-query -X avahi-libs
2020-01-29 16:27:54 bapt then I build a void * with all the indexes + integers
2020-01-29 16:28:08 bapt like <index of pkgname><index of pkg version> ...
2020-01-29 16:28:34 bapt I push that the same way (this is my actual package)
2020-01-29 16:29:04 bapt then I have a list of index of actual packages which I push as a key/value
2020-01-29 16:29:42 bapt so for a lookup I get they key and then loop over the packages quite quickly
2020-01-29 16:29:50 bapt not sure I am clear
2020-01-29 16:30:05 duncaen I think I understand
2020-01-29 16:30:28 bapt this is not really using key/value ;)
2020-01-29 16:30:36 anddam duncaen: I see, thanks
2020-01-29 16:30:54 duncaen yea makes sense that you don't really have to access package keys through the cdb
2020-01-29 16:31:06 bapt yeah
2020-01-29 16:31:34 bapt for the db of installed packages this approach won't work
2020-01-29 16:32:04 bapt note that I am not a key/value guy, I am discovering those tools
2020-01-29 16:32:12 bapt s/tools/libs/
2020-01-29 16:32:20 bapt so my approach may be brain damaged
2020-01-29 16:32:25 duncaen I've always wanted to not have a real database for installed packages, more like a state of packages in repositories, so that if you sync the repository, metadata could be updated without a real package update
2020-01-29 16:32:59 bapt not sure I follow you
2020-01-29 16:33:12 bin getting external info about software for the xbps pkg db
2020-01-29 16:33:35 duncaen so the idea would be that your repository will update package metadata, like conflicts and version requirements without having to rebuild the package
2020-01-29 16:34:06 bapt oh I would love that ;)
2020-01-29 16:34:15 duncaen lets say curl depends on libssl>=1.1.0, when libssl gets updated to an incompatible version, the curl dependency requirement gets changed to libssl=>1.1.0<1.2.0
2020-01-29 16:35:07 duncaen but I guess at some point you need a copy of the repository data anyways because we are going to drop package version users have installed so not sure how that should work
2020-01-29 16:35:18 bapt yeah I see, this is even more needed in my case, because version dependency is totally crap due to legacy in my package manager, so each minor version of a dependency bump requires a rebuild of the package
2020-01-29 16:35:25 bapt and I haven't managed to get rid of that legacy yet
2020-01-29 16:35:43 quinq Hi bapt :)
2020-01-29 16:35:49 bapt hey quinq
2020-01-29 16:36:40 bapt anyway for now I will just go back at stealing you work on libfetch ;)
2020-01-29 16:38:44 duncaen and I think I try to sync back some of your changes, we have some open issue about proxy environment variables without scheme because our parseURL is too strict
2020-01-29 16:39:38 duncaen oh by the way, the pthread stuff is in there because xbps-query -Ro is multi threaded lol
2020-01-29 16:39:58 duncaen otherwise it would be even slower
2020-01-29 16:39:58 bapt oh ;)
2020-01-29 16:40:21 bapt I have avoided thread in pkg, I fork when building a repo for speed but that is all
2020-01-29 16:41:13 duncaen we have a few multi threaded cases, like xbps-pkgdb -a, which checks checksums and some other things of installed packages
2020-01-29 16:42:13 bapt I find pkg check fast enough for now ;)
2020-01-29 16:42:26 bapt if uses complain maybe I will consider threading ;)