Skip to content

Internal Deploy Checks

Jump to bottom Edit New page
Ondrej Kosarko edited this page Oct 24, 2024 · 1 revision

Pre deploy check-list

Version: 2024/Sep/26

  • Verify BE is configured (clarin-dspace.cfg, dspace.cfg, submission-forms.xml, local.cfg with API KEYs)
  • Verify FE is configured (config.prod.yml)
  • Set up CRON jobs (dspace spring cron, crontab, and/or systemctl cron for letsencrypt)
  • The old database has been migrated to the new one
  • The old files (assetstore) have been copied
  • The data has been indexed after migration
  • The OAI-PMH has been indexed after migration
  • The administrator account has been created
  • The dspace UI has been manually tested
  • The backup has been configured (assetstore, volumes (database, solr, be), nginx - shibboleth (cfg, logs) - symlinks [ ] Shibboleth: Send Metadata.xml to IdP and/or federation e.g., eduID.cz
  • Shibboleth: Update shibboleth2.xml to allow wayf (e.g., eduID.cz wayf)
  • Shibboleth: To which group is assigned the user after shibboleth login?
  • Handle: Register a new handle prefix, generate sitebndl (IP and port of handle server) and send it to handle.net
  • Handle: Be sure that handle server is accessible from the outside
    • add simple www monitoring interface (in nginx.conf)
    stream {
    log_format bin '$remote_addr - - [$time_local] proto:$protocol $status bytes:$bytes_sent recv:$bytes_received time:$session_time "$upstream_addr"';
    # for handle server
    server {
        access_log /var/log/nginx/access-handleserver.log bin;
        error_log /var/log/nginx/error-handleserver.log;
  	    listen 2641;
          proxy_pass 127.0.0.1:2640;
    }
}
  • Handle: Update the docker-compose - handle port
  • Handle: check the info on a handle e.g., http://hdl.handle.net/11025/57420?noredirect&auth
  • Update iptables/ufw (probably you need to restart the docker service) 
    -A INPUT -p tcp --dport 443 -j ACCEPT
-A INPUT -p tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 2641 -j ACCEPT
-A INPUT -p udp --dport 2641 -j ACCEPT
  • Monitoring: Set up monitoring (Handle server, Shibboleth (DiscoFeed), BE, FE, SWORD?, OAI-PMH, SSL)
  • Ask for the DNS (alias, redirects, ...)
  • Generate SSL certificate for the new DNS
  • Change .env file with the new DNS
  • Set up the mail server and mail admin
  • Set docker containers to start automatically after reboot
  • Nginx: client_max_body_size
  • Set up system wide alert before going live with the new version
  • Check that the sword service is enabled
  • Redirects: OAI PMH proxy to BE instead of FE proxy (let nginx handle proxy) 
    location /oai {
    include proxy_params;
    proxy_pass https://xxx/server/oai;
  	break;
}
  • Redirects from old server to the new one: 
    server {
    server_name x.x.cz;
    listen 443 ssl;
    access_log /var/log/nginx/access.x.log;
    location /handle/ {
        return 301 https://x.x.cz$request_uri;
    }
    location /bitstream/ {
        return 301 https://dspace.x.cz$request_uri;
    }
    location / {
        return 301 https://dspace.x.cz/;
    }
    ssl_certificate /etc/letsencrypt/live/dspace5.x.cz/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/dspace5.x.cz/privkey.pem; # managed by Certbot
}
  • Prepare robots.txt - consider disallow / in the beginning
  • Check traffic on the old server
    • check old nginx/apache access logs e.g., for simple-search 
     location ~ ^/handle/.*/.*/simple-search {
     # Return a 403 Forbidden response
     access_log /var/log/nginx/access.403.simple.search.log;
     return 403;
 }
  • The images in the UI cannot be big
  • Set up API-KEYs (google.analytics, sherpa.romeo.apikey, sherpa.romeo.url)
  • Solr reindex sequence - When the user add a new Item the solr statistics are updated every 10minutes - is it OK? If not, change it here: https://github.com/dataquest-dev/DSpace/blob/dtq-dev/dspace/solr/statistics/conf/solrconfig.xml#L53
  • Has the customer some special group of users? E.g., the user could see the bitstream, but not the metadata
  • DO NOT USE localhost use 127.0.0.1 because of ipv6
  • docker-compose volume must point to existing file because it creates a new one folder
  • How often will be the logs rotated? (converting to gzip)

CRON Jobs example

30 22 * * * cd /app && ./run-cli-command filter-media

0 23 * * * cd /app && ./run-cli-command oai import

0 0 * * * cd /app && ./run-cli-command index-discovery

0 2 * * * cd /app && ./run-cli-command stats-util -i (This removes any known web spiders from your usage statistics)

1 3 * * * cd /app && ./run-cli-command subscription-send -f D

2 3 * * 0 cd /app && ./run-cli-command subscription-send -f W

3 3 1 * * cd /app && ./run-cli-command subscription-send -f M

0 4 1 * * cd /app && ./run-cli-command cleanup

Post deploy check-list

Version: 2024/Sep/26

This is a list that should be checked after new deploy to ensure some of the functionality and stability

  • Nginx proxy timeouts and caches 
    proxy_connect timeout 60;
proxy_send_timeout 60;
proxy_read_timeout 60;
send_timeout 60;
  • Nginx max filesize for upload, set with nginx client_max_body_size 10G;
  • Resources
    • CPU - a lot will be taken by angular, especially in the beginning when caching, but several 10s of percent should remain free (so at most 60, 70% utilization of CPU)
    • RAM - depending on number of nodes in pm2, a lot of RAM can be taken, but some needs to be free.
    • swapping - make sure to create a lot of swap space, just in case of some more RAM-intensive operations as a backup solution
  • Filesystem size - very individual, depending on how many files will be stored, but perhaps 1 or more TBs
  • Network speed - besides the obvious (load speed), uploading big files (especially with S3 enabled) can be slowed down, as well as calculating checksums etc.
  • Check all occurences of lindat.mff.cuni.cz url and update/change it
  • Check if license does not contain XXX e.g. http://dev-5.pc:85/XXX/static/
  • inlineCriticalCss in both the environment.prod.ts file is set to false and in config.prod.yml
  • Generate sitemap_index.xml - in the BE running the specific dspace/bin/dspace command
    • robots.txt -> sitemap_index.xml, should be available once online
    • consider copying robots.txt and serve it from nginx (can be changed easily e.g., disable all bots at the beginning)
    # 2024/08 from docker-ui
location = /robots.txt {
    alias /app/html/robots.txt;
}
  • Check if the robots.txt is set up correctly - there could be problem with docker mounting
    • Consider nginx proxy to avoid going to FE that transparently streams from BE
    # sitemaps are big, fetch them using nginx rather than node
location ~ ^/sitemap(.*) {
    include proxy_params;
    proxy_pass https://x.x.cz/server/sitemaps/sitemap$1;
    break;
}
  • check GZIP compression is enabled in nginx.conf 
    gzip on;
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
  • Check SWAP in the system - Do we want it? How much?
  • Set up nodes in the FE following the current number of processors
    • dspace-ui.json
    {
  "apps": [
    {
      "name": "dspace-ui",
      "cwd": "/app",
      "script": "dist/server/main.js",
      "time": true,
      "instances": "9",
      "exec_mode": "cluster",
      "node_args": "--max_old_space_size=1344",
      "max_memory_restart": "2G",
      "env": {"NODE_ENV": "production"}
    }
  ]
}
  • Check netids of migrated users, very likely they will have to be erased
  • The community logos are set up
  • Send monitoring endpoints
  • Send an email to check google statistics
Add a custom footer
Add a custom sidebar
Clone this wiki locally