'm concerned that there are nightly builds available on download.java.net that point to a repository that doesn't have the packages that were used to create the build. That seems to be a problem. If the nightly builds are made available for download, then there should be a nightly repository too. With the current way the Catalog class is structure, there isn't really any good way to bring in the Fmri's for the installed packages. The CatalogCache I'm adding for the multi-repo feature will allow that. Calling catalog.refresh is just one of many things that bootstrap of updatetool does. But WRT this behavior, calling just catalog.refresh would produce the same results. A catalog.refresh downloads the complete catalog via HTTP, stores it in a file, and then loads it into memory. Right now, the pkg.glassfish.org catalog for v3prelude is 5380 bytes. It also wipes out the catalog cache that the python CLI maintains, so it will need to rebuild it the next time it runs. Tom Anissa Lam wrote: > Hi Tom, > > Thanks for the detailed explanation. It is a known fact that only packages from promoted build is pushed to the UC repository, thats why the installed packages from nightly build is missing from the repo. > As you mentioned, the behavior of Java API and pkg CLI is different, and it can be very confusing to user. We know about this root cause, but not everyone does. Will it be possible to fix it now so that they behave the same instead of having to wait for the support of multi-repo feature ? > > Also, is calling catalog.refresh() gives the same result as doing the bootstrap of updatetool ? How expensive is a catalog.refresh() call is ? > > thanks > Anissa > > > Tom Mueller wrote: >> The root cause of this problem is that the packages that you have installed in the image are not in the repository that the image points to. For example, the following felix package is installed: >> >> felix@1.0.4%2C0-0%3A20080903T163046Z >> >> but the following felix packages are in the repository: >> >> V pkg felix 1.0.4,0-0:20080816T220653Z >> V pkg felix 1.0.4,0-0:20080820T134004Z >> V pkg felix 1.0.4,0-0:20080828T213242Z >> V pkg felix 1.0.4,0-0:20080830T023728Z >> >> Note that there is no felix package in the repository with a 20080903 timestamp. >> >> One difference between the Java API and the pkg CLI is that the Java API ignores any packages that are installed but do not show up in the catalog from the repository. The pkg CLI will recognize packages that are installed but not in the catalog. As part of the multi-repo support, I was planning to bring in the code that fixes this. >> >> The reason why the Java API works before you do the bootstrap is that the catalog in the image has not yet been refreshed from the repository. One of the things the bootstrap does is to do a catalog refresh. >> >> Tom >> >> Anissa Lam wrote: >>> Hi Tom, >>> >>> I just tested that my experience is the same with nightly build as well. >>> This is the nightly build, which is today's build that i am using. >>> >>> http://download.java.net/glassfish/v3/nightly/glassfish-snapshot-v3-prelude-09_03_2008.zip >>> >>> Thanks for looking into this. >>> Anissa >>> >>> Tom Mueller wrote: >>>> Anissa, >>>> Can you send me the gfv3-prelude-ips.zip file? >>>> Thanks. >>>> Tom >>>> >>>> Anissa Lam wrote: >>>>> >>>>> Tom, >>>>> Not sure if this is due to the fact that the server is built by myself, but i am seeing the issue when i did exactly what you did. >>>>> Snjezana, do you know if there is problem when i unzip distributions-prelude/gfv3-prelude/target/gfv3-prelude-ips.zip to setup my env instead of using a promoted build ? >>>>> The manifest file of pkg-client.jar says: >>>>> >>>>> Bundle-SymbolicName: com.sun.pkg.client >>>>> Bundle-Name: pkg-client >>>>> Bundle-Version: 1.0.7.1116 >>>>> Bundle-ManifestVersion: 2 >>>>> >>>>> and the console is using pkg-client-1.0.7-14.1120.jar >>>>> >>>>> >>>>> Here is the script from my terminal. >>>>> >>>>> macau2:~/Awork/V3/v3/glassfishv3-prelude anilam$ java -jar pkg/lib/pkg-client.jar list >>>>> Sep 3, 2008 9:27:22 AM com.sun.pkg.client.Image >>>>> INFO: Software repository: http://pkg.glassfish.org/dev/mac/v3prelude/ >>>>> pkg:/pkg-java@1.0.7,0-14.1116:20080820T191512Z >>>>> pkg:/felix@1.0.4,0-0:20080901T075521Z >>>>> pkg:/glassfish-hk2@3.0,0-23:20080901T075642Z >>>>> pkg:/glassfish-grizzly@3.0,0-23:20080901T075712Z >>>>> pkg:/glassfish-api@3.0,0-23:20080901T075900Z >>>>> pkg:/glassfish-nucleus@3.0,0-23:20080901T080040Z >>>>> pkg:/glassfish-common@3.0,0-23:20080901T080205Z >>>>> pkg:/glassfish-registration@3.0,0-23:20080901T080230Z >>>>> pkg:/glassfish-management@3.0,0-23:20080901T080300Z >>>>> pkg:/glassfish-amx@3.0,0-23:20080901T080348Z >>>>> pkg:/glassfish-jca@3.0,0-23:20080901T080408Z >>>>> pkg:/glassfish-jpa@3.0,0-23:20080901T080547Z >>>>> pkg:/glassfish-jta@3.0,0-23:20080901T080600Z >>>>> pkg:/glassfish-jsf@3.0,0-23:20080901T080627Z >>>>> pkg:/glassfish-web-management@3.0,0-23:20080901T080640Z >>>>> pkg:/glassfish-web@3.0,0-23:20080901T080828Z >>>>> pkg:/glassfish-jdbc-management@3.0,0-23:20080901T080846Z >>>>> pkg:/glassfish-jdbc@3.0,0-23:20080901T080901Z >>>>> pkg:/glassfish-gui@3.0,0-23:20080901T080943Z >>>>> pkg:/glassfish-web-gui@3.0,0-23:20080901T080950Z >>>>> pkg:/glassfish-jdbc-gui@3.0,0-23:20080901T080954Z >>>>> pkg:/glassfish-scripting@3.0,0-23:20080901T081027Z >>>>> pkg:/javadb@10.2.2.1,0-0:20080901T081102Z >>>>> macau2:~/Awork/V3/v3/glassfishv3-prelude anilam$ java -jar pkg/lib/pkg-client.jar list | wc >>>>> Sep 3, 2008 9:27:36 AM com.sun.pkg.client.Image >>>>> INFO: Software repository: http://pkg.glassfish.org/dev/mac/v3prelude/ >>>>> 23 23 1093 >>>>> macau2:~/Awork/V3/v3/glassfishv3-prelude anilam$ >>>>> macau2:~/Awork/V3/v3/glassfishv3-prelude anilam$ bin/updatetool >>>>> >>>>> The software needed for this command (updatetool) is not installed. >>>>> >>>>> If you choose to install Update Tool, your system will be automatically >>>>> configured to periodically check for software updates. If you would like >>>>> to configure the tool to not check for updates, you can override the >>>>> default behavior via the tool's Preferences facility. >>>>> >>>>> When this tool interacts with package repositories, some system information >>>>> such as your system's IP address and operating system type and version >>>>> is sent to the repository server. For more information please see: >>>>> >>>>> http://wiki.updatecenter.java.net/Wiki.jsp?page=UsageMetricsUC2 >>>>> >>>>> Once installation is complete you may re-run this command. >>>>> >>>>> Would you like to install Update Tool now (y/n): y >>>>> >>>>> Software repository: http://pkg.glassfish.org/dev/mac/v3prelude/ >>>>> Install image: /Users/anilam/Awork/V3/v3/glassfishv3-prelude/bin/.. >>>>> Installing pkg packages. >>>>> pkg:/pkg@1.0.7,0-14.1116:20080820T193330Z: downloading manifest from server >>>>> pkg:/python2.4-minimal@2.4.4.0,0-14.1116:20080820T193342Z: downloading manifest from server >>>>> Installing: [pkg:/pkg@1.0.7,0-14.1116:20080820T193330Z] >>>>> pkg:/pkg@1.0.7,0-14.1116:20080820T193330Z: downloading 205 files >>>>> pkg:/python2.4-minimal@2.4.4.0,0-14.1116:20080820T193342Z: downloading 291 files >>>>> Installing updatetool packages. >>>>> pkg:/updatetool@2.0.0,0-14.1116:20080820T193412Z: downloading manifest from server >>>>> pkg:/wxpython2.8-minimal@2.8.8,0-14.1116:20080820T193428Z: downloading manifest from server >>>>> Installing: [pkg:/updatetool@2.0.0,0-14.1116:20080820T193412Z] >>>>> pkg:/updatetool@2.0.0,0-14.1116:20080820T193412Z: downloading 302 files >>>>> pkg:/wxpython2.8-minimal@2.8.8,0-14.1116:20080820T193428Z: downloading 296 files >>>>> Registering notifier: Already registered. >>>>> Initialization complete. >>>>> >>>>> Software successfully installed. You may now re-run this command (updatetool). >>>>> macau2:~/Awork/V3/v3/glassfishv3-prelude anilam$ java -jar pkg/lib/pkg-client.jar list | wc >>>>> Sep 3, 2008 9:34:40 AM com.sun.pkg.client.Image >>>>> INFO: Software repository: http://pkg.glassfish.org/dev/mac/v3prelude/ >>>>> 5 5 254 >>>>> macau2:~/Awork/V3/v3/glassfishv3-prelude anilam$ java -jar pkg/lib/pkg-client.jar list >>>>> Sep 3, 2008 9:39:11 AM com.sun.pkg.client.Image >>>>> INFO: Software repository: http://pkg.glassfish.org/dev/mac/v3prelude/ >>>>> pkg:/pkg@1.0.7,0-14.1116:20080820T193330Z >>>>> pkg:/pkg-java@1.0.7,0-14.1116:20080820T191512Z >>>>> pkg:/python2.4-minimal@2.4.4.0,0-14.1116:20080820T193342Z >>>>> pkg:/updatetool@2.0.0,0-14.1116:20080820T193412Z >>>>> pkg:/wxpython2.8-minimal@2.8.8,0-14.1116:20080820T193428Z >>>>> macau2:~/Awork/V3/v3/glassfishv3-prelude anilam$ >>>>> >>>>> >>>>> thanks >>>>> Anissa. >>>>> >>>>> Tom Mueller wrote: >>>>>> Anissa, >>>>>> >>>>>> After updatetool or the CLI does anything that modifies the image, it is necessary to create a new Image object in the Java API. The Image object should not be kept around indefinitely within the JVM. In order to pick up changes in the image on disk, the Image object needs to be recreated. >>>>>> >>>>>> It is not necessary to call Image.refreshCatalog (or Catalog.refresh which is the same thing) unless the catalog in the repository has changed. >>>>>> >>>>>> The behavior after the server restart sounds like a bug. >>>>>> >>>>>> I tried recreating it with the following test case but I'm not seeing the problem. Here's what I did: >>>>>> >>>>>> 1. unzip gfv2.zip from the Try It page into a directory. >>>>>> 2. cd to that directory >>>>>> 3. java -jar pkg/lib/pkg-client.jar list - this shows the pre-installed packages: ant, pkg-java, updatetool-data >>>>>> 4. run bin/pkg - this installs pkg/min python >>>>>> 5. do step 3 again - this shows all of the packages correctly >>>>>> 6. install updatetool using python IPS: bin/pkg install updatetool >>>>>> 7. do step 3 again - still shows all of the packages correctly. >>>>>> >>>>>> Any idea how this test case is different from what you are doing? >>>>>> >>>>>> Note: the "list" subcommand on pkg-client.jar does an Image.getInventory(null, false) >>>>>> >>>>>> Tom >>>>>> >>>>>> >>>>>> Anissa Lam wrote: >>>>>>> Hi Tom, >>>>>>> I found a very strange behavior and not sure if this is a known issue. >>>>>>> >>>>>>> - Start up admin console, it showed correctly all the installed pkg and Add-ons. >>>>>>> >>>>>>> - Start up updatetool, since this is the first time, it asked me if i want to download the pkgs needed for updatetool. I confirmed and then i restart to see updatetool. >>>>>>> >>>>>>> - Go back to admin console, and uses the same API to get the list of installed component, it doesn't show those related to update tool, eg python2.4-minimal, etc. >>>>>>> question: am i suppose to do image.refreshCatalog() everytime I try to get the list , how about catalog.refresh() ? >>>>>>> >>>>>>> - I restart the server. then go to console again, this time the list of installed components only shows those related to updatetool. All the GlassFish packages now shows as Available Add-Ons. >>>>>>> >>>>>>> Can you help explain what i am seeing ? Should i file a bug on this ? >>>>>>> >>>>>>> thanks >>>>>>> Anissa. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>>