CDS

Make your HiPS in 10 steps

By P.Fernique (last update July 2022 - additions in green)

... or how to build a Hierarchical Progressive Survey from your image (or cube) collection.

Your HiPS step by step

  1. Download the latest version of Aladin/Hipsgen code. Do not hesitate to use the last beta version:
    → AladinBeta.jar
    Note: You can also use Hipsgen.jar package, but it is more useful to use directly Aladin for generating & checking the HiPS with the same tool (and concretely, it is the same code).
  2. Set together the collection of the original images or cubes in a local directory (sub-directory hierarchy is allowed, symbolic links can be useful, gzipped ou FITS compressed images (rice, ...) are allowed)
    → we assume "Data" directory
    Note: There is no limit (image size and/or number of images) apart your disk.
  3. Optional: Load one image in Aladin to be sure that its astrometrical calibration is correctly interpreted and to determine the HiPS generation parameters:
    → java -Xmx2g -jar AladinBeta.jar Data/OneImage.fits
    • If your images are packaged in Multi-Extension FITS, memorize the index of the good HDU or possibly all HDUs for multi-CCD packaging (cf. hdu parameter)
    • Adjust the contrast and memorize the pixel min & pixel max display value (Ctrl+M: left and right values over the pixel histogram) (cf. hips_pixel_cut parameter)
    • Check if the borders of your images are ok, or if you have to remove a margin, and/or if the borders of image have not been observed -typically filled up with 0 (cf. blank parameter), or if your observations are circular (cf. shape parameter).
    • Look in the FITS header (Alt+H) which additionnal keywords could be kept as meta data information for progenitor facility (see below), for instance: DATE-OBS, INSTRUM, EXP_TIME, ... (cf. fitskey parameter)
  4. Optional: Create a "pilot" HiPS on few images for a quick test:
    • Launch the Pilot HiPS creation on the first 100 images by this script command:
      → java -Xmx16g -jar AladinBeta.jar -hipsgen in=Data out=PilotHiPS creator_did=HiPSID pilot=100
    • Check the resulting HiPS thanks to Aladin:
      → java -jar AladinBeta.jar PilotHiPS
    • If required, reiterate the HiPS computation by inserting/modifying these Hipsgen parameters: blank, border, fov, skyval, hips_pixel_cut, fading, mixing, partitioning... - see the manual below.
      Do not forget to insert "-f" option for removing the previous HiPS (or remove it manually).
  5. Launch the HiPS creation on all original data.
    Take care that this process can be long and very computer consuming (RAM and CPU). You can adjust the RAM by the java -XmxNNg parameter, and the number of threads by the maxthread Hipsgen parameter. The final HiPS size can be estimated to 1.3x of the original size of data.
    → java -Xmx16g -jar AladinBeta.jar -hipsgen maxthread=20 in=Data ...
  6. Check the result thanks to Aladin Desktop and/or via Aladin Lite
    → Aladin Desktop: java -jar AladinBeta.jar DataHiPS
    → Aladin Lite: in your browser, load the directory DataHiPS
  7. Complete the description of your HiPS by editing the file DataHiPS/properties (read the inside comments). Do not forget to provide a creator_did identifier for your HiPS. The syntax follows the IVOA IVOID recommendation (ivo://XXX/...). The first word MUST be the acronym of your institute (CDS, IAS, ESAC, STScI, ...) and your HiPS - or atleast your institute - should be declared in the IVOA VO registry (see point 10).
  8. Distribute your HiPS by serving or moving your DataHiPS directory thanks to a basic httpd server (apache...).
    No data base software or CGI are required, just a HTTP access to the HiPS directory.
  9. Optional: Add your HiPS in Aladin and/or other HiPS aware clients.
    For that, you need to declare your "HiPS server" and become by this way a member of the "HiPS network".
    • Become a official HiPS server by just generating and publishing your hips list. The HiPS list is basically the concatenation of all the properties files of your HiPS. You can do that manually, or via a CGI script like this one (cf HiPS IVOA doc - section 5, page 22)
    • Declare your HiPS server in the VO registry (see IVOA document section 5.3.1 page 24). In parallel, you can contact the CDS which temporary maintains the list of HiPS servers.
    • Check if your HiPS is now visible in the HiPS clients of your choice. If it is not the case, contact directly the HiPS clients developers (CDS for Aladin Desktop or Aladin Lite). For your help, the CDS maintains this HiPS table which is aggregating all declared HiPS lists.
  10. Optional: Declare your individual HiPS in the VO registry
    (see IVOA document - section 5.3.2 page 25)

A few additional tips

  1. The observations do not fully cover the whole image arrays (ex: HST, Chandra):
    • If your image contains not exposed pixels, check if the BLANK FITS keyword is correctly set to the value of these pixels. If it is not the case, specify it via the parameter blank=xx (generally blank=0). Notice that the BLANK value is expressed as the FITS pixel coding (BSCALE and BZERO not applied), and not in physical quantity.
    • Check if there are bad pixels at the borders of the image. You can remove them via the parameter border=nn.
    • If the observation is circular or rectangular (ex: GALEX), you can specify it via the parameter shape=sss
    • If the observation is complex and can be discribed by a polygon (ex: Schmidt plate), you can specify a polygon, either associated to a specifical image (same name, .fov extension), or for a collection of images set together in a sub-directory - the file describing the polygon must have the same name of the directory, .fov extension. You must use fov=true parameter to take into account these polygon files.
      Tips:Use Aladin Desktop with the tool Tag for generating the list of coordinates. Cut-&-paste the result in a clip board for keeping only the RA and DEC columns (one per line).
  2. The sky back grounds of the observations are not homogeneous:
    • If the sky back ground value is specified in the FITS header, you can indicate the corresponding FITS key to Aladin/Hipsgen via the parameter skyval=XXXX (for instance skyval=BACKGRND)
    • Otherwise, you can try with skyval=true for asking Aladin/Hipsgen to compute an estimation of each sky back ground.
  3. The original images are packaged in Multi-Extension FITS:
    Specify the good extension via the parameter hdu=nn, or all hdu=all.
  4. You would like to modify the BITPIX, notably if the original images are coded in a surprising bit range (ex: -64 for counting a few photons...) and/or you would like to minimize the HiPS size:
    • Choose a good target BITPIX (ex: 16 means 2^16 = 65536 grey levels typically enough for most of detectors)
    • Specify this target BITPIX via the parameter bitpix=nn.
    • Determine manually the physical range of original valid pixels and specify them thanks to pixelRange="min max" parameter. Otherwise Aladin/Hipsgen will do an automatical estimation on the first image of the collection, or on a specifical image specified by img="path" parameter. The min and max values are expressed in physical quantities (BSCALE, BZERO applied).
  5. Your image collection are in JPEG or PNG, and not in FITS:
    • For each image, you must provided the associated astrometrical calibration - AVM tags or WCS solution - either directly inside the JPEG (resp. PNG) comment segment, or for WCS solution, in a separated file same name, .hhh extension) or . The WCS solution can be expressed as a FITS header (80 character alignement), or with basic ASCII line separators.
    • Specify the resulting tile format via the parameter color=jpeg or color=png (or color=true to keep the same format that the original images).
    Tips: If your JPEG or PNG images are quite huge (>100 Megapixels), do not hesitate to split your images in sub-images of 1024x1024 pixels. The process will be really faster. Do not forget to adjust the WCS solutions according to.
  6. Your image is a HEALPix fits map:
    In this case, your original data will be not resampled, but just mapped into HiPS tiles. You have to specify the filename of the HEALPix map instead of the input directory:
    →java -Xmx2000m -jar Aladin.jar in="YourHealpixMap.fits
  7. You would like to restart partially the process:
    • You can specify on which region you want to restart the process. Use the parameter moc=sss where sss is a inline ASCII MOC or a path to a MOC file.
      Have a look on mode=xxx parameter for controlling the pixels and tiles merging/replacing behaviour...
    • You can also launch Aladin/Hipsgen step by step by specifying explicitely each action. The default action list is INDEX TILES PNG CHECKCODE DETAILS UPDATECHECKCODE. You can restart one specifical action, for instance relaunch PNG tile computation
      →java -jar AladinBeta.jar -hipsgen in=Data PNG
  8. You would like to generate a colored HiPS combining several survey filters (2 or 3):
    • For each filter, you have to create before the grey level HiPS.
      → we assume "redHips", "greenHips" and "blueHips" paths.
    • After that, launch Aladin/Hipsgen RGB action with these parameters:
      →java -jar AladinBeta.jar -hipsgen inRed=redHips inGreen=greenHips inBlue=blueHips RGB
      You can adjust the color mapping scale for each color component thanks to cmRed, cmBlue and cmBlue parameters (see details below)
    • From Aladin version 11.9 onwards, it is possible to view the expected HiPS colour result before calculation and to retrieve the parameters for the corresponding Hipsgen command.
  9. You would like to generate your HiPS incrementally
    This facility is useful for incorporating new observations in an already existing HiPS without restarting all the process (or if you have no longer access to the original images). This option is only supported in FITS mode.
    • Your original HiPS had to be generated with the -live parameter
      →java -jar AladinBeta.jar -hipsgen -live in=... out=DataHiPS ...
    • Append new observations thanks to APPEND action:
      →java -jar AladinBeta.jar -hipsgen in=... out=DataHiPS APPEND
    • Note that incremental addition will require twice the disk space (storage of weight tiles). Use only if necessary. It is also possible to delete the weight tiles after the process is finished (=> action CLEANWEIGHT)
  10. You would like to concatenate two HiPS
    This functionality is possible between two compatible HiPS (same coordinate system, same order, same format, same settings for FITS tiles). The best results will be obtained in FITS mode with the -live option.
    • Generate the first HiPS:
      →java -jar AladinBeta.jar -hipsgen in=... out=Hips1 ...
    • Generate the second HiPS:
      →java -jar AladinBeta.jar -hipsgen in=... out=Hips2 ...
    • Concatenate the second HiPS into the first one thanks to CONCAT action:
      →java -jar AladinBeta.jar -hipsgen in=Hips2 out=Hips1 CONCAT
  11. You would like to patch (or regenerate) a HiPS on a specific region
    You can restrict the HiPS target size thanks to the region=moc parameter. This "MOC" can be described via an inline string, or via a real MOC file.
    Associated to the appropriated mode parameter, notably the value OVERWRITE (pixel replacement), or KEEP (pixel completion), you will have the full control to decide how to patch an already existing HiPS.
    → java -jar AladinBeta.jar -hipsgen in=Data region="3/131,132 4/565" mode=OVERWRITE ...
    Note: For advanced mode parameters (REPLACETILE, KEEPTILE) take care that you need to have access to all the original images ("in" directory must be unchanged). If you want to add new observations see the previous item)
  12. You would like to generate the progenitors links:
    This facility generates links for providing an access to the original images.
    • Launch Aladin/Hipsgen DETAILS action if required (already done by default)
    • Edit the file DataHiPS/HpxFinder/metadata.xml and follow the inside explanations, notably for re-incorporating the metadata memorized thanks to fitskey="xxx" parameter.
  13. You want to clone a remote HiPS:
    This feature allows you to mirror a HiPS. Be sure to check the copy and redistribution rights beforehand.
    Launch Aladin/Hipsgen with MIRROR action:
    →java -jar AladinBeta.jar -hipsgen in=http://remote/url MIRROR
    Note: The resulting HiPS will be stored in the current directory, and the metadata will be automatically adapted for describing a "mirrored" HiPS.

For boosting the process

  1. If the images are relatively small (< 3 Megapixels) and the collection does not have a lot of overlays (<6 images on the same location), set the parameter: partitioning=false.
  2. If the images have already homogeneous sky back ground, set the parameter: mixing=false.
  3. If your have to generate JPEG or PNG tiles:
    • JPEG is 4 to 20 times faster than PNG (but does not manage transparency and introduces artefacts).
    • PNG encoder time can vary by a factor 10 depending of the sky background noise. More you remove the sky background noise by adjusting the min pixel cut, faster the PNG encoder will work.
  4. For checking the result before the end it is possible to have a quick look of the current result:
    1. Stop Aladin/Hipsgen: kill -STOP pid, or CTRL+Z
    2. Launch this command: java -jar AladinBeta.jar -hipsgen in=Data ALLSKY
    3. Look the current result: java -jar AladinBeta.jar DataHiPS
    4. Restart the process: kill -CONT pid, or fg

Manual v11.915

Usage: java -jar HipsGen in=file|dir [otherParams ... ACTIONs ...]
       java -jar HipsGen -param=configfile

The config file must contain these following options, or use them
directly on the comand line :


Available actions (by default: "INDEX TILES PNG CHECKCODE DETAILS"):
   INDEX      Build spatial index (in HpxFinder directory) + MOC index
   TILES      Build all true value pixel tiles (FITS) + Allsky.fits + MOC
   JPEG       Build all preview tiles (JPEG) + Allsky.jpg
   PNG        Build all preview tiles (PNG) + Allsky.png
   CHECKCODE  Compute+store the check codes (and the size) associated to the target HiPS
   RGB        Build and RGB HiPS based on 2 or 3 other HiPS
   MOC        (Re)build the MOC (MultiOrder Coverage map)
   ALLSKY     (Re)build all Allsky files + index.html
   TREE       (Re)build HiPS tree structure from already existing tiles
   MAPTILES   Build all FITS tiles from a HEALPix Fits map
   APPEND     Append new images/cubes to an already existing HiPS
   CONCAT     Concatenate one HiPS to another HiPS
   CUBE       Create a HiPS cube based on a list of HiPS (; separated)
   CLEANFITS  Delete all FITS tiles and Allsky.fits
   DETAILS    Adapt HiPS index for supporting the "detail table" facility
   MAP        Build an HEALPix map from the HiPS tiles
   MIRROR     Mirror a remote HiPS locally
   UPDATE     Upgrade HiPS metadata additionnal files to last HiPS standard
   CHECK      Verify the HiPS based on the check codes previously computed (CHECKCODE)
   CHECKDATASUM  Verify the FITS tiles based on internal FITS DATASUM and the check code
   LINT       Check HiPS IVOA 1.0 standard compatibility

Required parameter:
   in=dir                  Source image directory (FITS or JPEG|PNG +hhh or HiPS),
                           unique image or HEALPix map file
   creator_did=id      HiPS identifier (syntax: AUTHORITY/internalID)

Basic optional parameters:
   out=dir             HiPS target directory (default ./+"AUTHORITY_internalID")
   hips_creator=name   Name of the person|institute who builds the HiPS
   hdu=n1,n2-n3,...|all List of HDU numbers (0 is the primary HDU - default is 0)
   blank=nn|key        Specifical BLANK value, or alternate BLANK fits keyword
   color=jpeg|png      The source images are colored images (jpg or png) and the tiles
                       will be produced in jpeg (resp. png)
   -n                  Just print process information, but do not execute it.
   -f                  clear previous computations

Advanced optional parameters:
   hips_order=nn       Specifical HEALPix order - by default, adapted to the original
                       resolution
   hips_pixel_bitpix=nn Specifical target bitpix (-64|-32|8|16|32|64)
   hips_pixel_cut=min max Specifical pixel cut and/or transfert function for PNG/JPEG 8 bits
                       conversion - ex: "120 140 log")
   hips_data_range=min max Specifical pixel value range (required for bitpix
                       conversion, or for removing bad pixels - ex: "-5 110")
   img=file            Specifical reference image for default initializations 
                       (BITPIX,BSCALE,BZERO,BLANK,order,pixelCut,pixelRange)
   mode=xx             Coadd mode, action dependent:
                       .TILES (restart) -> pixel impact: *OVERWRITE*|KEEP|SUM|AVERAGE
                                  tile impact: REPLACETILE|KEEPTILE
                       .CUBE   -> tile copy: LINK|*COPY*
                       .CONCAT -> pixel impact: OVERWRITE|KEEP|ADD|DIV|MUL|*AVERAGE*
                                  tile impact : REPLACETILE|KEEPTILE
   partitioning=true|false|nnn True for cutting large original images in blocks of nnn x nnn 
                       (default is true, nnn=512 )
   region=moc          Specifical MOC region to compute (ex: 3/34-38 50 53)
                       or Moc.fits file (all sky by default)
   fov=true|x1,y1..    Observed regions by files.fov or global polygon (in FITS convention).
   hips_min_order=nn   Specifical HEALPix min order (only for DETAILS action)
   hips_frame          Target coordinate frame (equatorial|galactic)
   hips_tile_width=nn  Specifical tile width (pow of 2) - default 512
   hips_status=xx      HiPS status (private|public clonable|clonableOnce|unclonable)
                       (default: public clonableOnce)
   cache=dir           Directory name for an alternative cache disk location
   cacheSize=nn        Alternative cache disk size limit (in MB - default 1024
   cacheRemoveOnExit=true|false Remove or not the cache disk at the end - default true
   maxThread=nn        Max number of computing threads
   target=ra +dec      Default HiPS target (ICRS deg)
   targetRadius=rad    Default HiPS radius view (deg)
   pilot=nnn           Pilot test limited to the nnn first original images.
   verbose=n           Debug information from -1 (nothing) to 4 (a lot)
   -live               incremental HiPS (keep weight associated to each HiPS pixel)
   -notouch            Do not touch the hips_release_date
   -color              Colorized console log messages

Specific optional parameters:
.INDEX action:
   fitskeys=list       Fits key list (blank separator) designing metadata FITS keyword value 
                       to memorized in the HiPS index
.TILES action:
   shape=...           Shape of the observations (ellipse|rectangle)
   border=...          Margins (in pixels) to ignore in the original observations
                       (top left bottom right or constant)
   skyval=key|auto|%info|%min %max   Fits key to use for removing a sky background, or auto
                       detection or percents of pixel histogram kept (central ex 99, or
                       min max ex 0.3 99.7)
   exptime=key         Fits key to use for adjusting variation of exposition
   fading=true|false   False to avoid fading effect on overlapping original images 
                       (default is false)
   mixing=true|false   False to avoid mixing effect on overlapping original
                       images (default is true [pixel average])
   pixelGood=min [max] Range of pixel values kept
   maxRatio=nn         Max height/width pixel ratio tolerated for original obs 
                       (default 2, 0 for removing the test)
   method=m            Method (MEDIAN|MEAN|FIRST) (default MEDIAN) for aggregating colored 
                       compressed tiles (JPEG|PNG)
.RGB action:
   inRed               HiPS red path component, possibly suffixed by cube index (ex: [1])
   inGreen             HiPS green path component, possibly suffixed by cube index (ex: [1])
   inBlue              HiPS blue path component, possibly suffixed by cube index (ex: [1])
   cmRed               Colormap parameters for HiPS red component (min [mid] max [fct])
   cmGreen             Colormap parameters for HiPS green component (min [mid] max [fct])
   cmBlue              Colormap parameters for HiPS blue component (min [mid] max [fct])
   luptonQ=x           Q coef Lupton RGB builder (default auto)
   luptonS=x/x/x       scale coefs Lupton RGB builder (default auto)
   luptonM=x/x/x       m coefs Lupton RGB builder (default auto)
   filter=gauss        Gaussian filter applied on the 3 input HiPS

.MOC action:
   mocOrder=s          Specifical MOC order.
                       s-spaceOrder, t-timeOrder, maxLimit, degradation rule
.MAP action:
   nside=nn            HEALPix map NSIDE - by default 2048

.MIRROR action:
   tileTypes           List of tile format to copy
   split='size;altPath]' multi disk partition split (ex: 300g;/hips/part2)
   -nocheck            Do not check date&size of local tiles
   -nice               Slow download for avoiding to overload remote http server

© Université de Strasbourg/CNRS

    • Contact