add horizontal file sharding based on filename matches

Docker/Readme.md

@@ -12,18 +12,19 @@ It is possible to build all required images at once by running `docker-build.bat
 
 ## 2. Configure ports + token
 You should set up 2 environment variables that hold server specific configuration and open up ports.  
-The default ports used through the provided configuration are `6000` for the main server and `6200` for the files downloads.  
+The default ports used through the provided configuration are `6000` for the main server and `6200` as well as `6201` for the files downloads.
 Both ports should be open to your computer through your router if you wish to test this with clients.
 
 Furthermore there are two environment variables `DEV_MARE_CDNURL` and `DEV_MARE_DISCORDTOKEN` which you are required to set.  
-`DEV_MARE_CDNURL` should point to `http://<yourip or dyndns>:6200/cache/` and `DEV_MARE_DISCORDTOKEN` is an oauth token from a bot you need to create through the Discord bot portal.  
+`DEV_MARE_CDNURL` should point to `http://<yourip or dyndns>:6200/cache/` and `DEV_MARE_DISCORDTOKEN` is an oauth token from a bot you need to create through the Discord bot portal. 
+You should also set `DEV_MARE_CDNURL2` to `http://<yourip or dyndns>:6201/cache/`
 It is enough to set them as User variables. The compose files refer to those environment variables to overwrite configuration settings for the Server and Services to set those respective values.  
 It is also possible to set those values in the configuration.json files themselves.  
 Without a valid Discord bot you will not be able to register accounts without fumbling around in the PostgreSQL database.
 
 ## 3. Run Mare Server
 The run folder contains two major Mare configurations which is `standalone` and `sharded`.  
-Both configurations default to port `6000` for the main server connection and `6200` for the files downloads. No HTTPS.  
+Both configurations default to port `6000` for the main server connection and `6200` for the files downloads. Sharded configuration additionally uses `6201` for downloads. No HTTPS.  
 All `appsettings.json` configurations provided are extensive at the point of writing, note the differences between the shard configurations and the main servers respectively.  
 They can be used as examples if you want to spin up your own servers otherwise.
 

Docker/run/compose/mare-sharded.yml

@@ -20,7 +20,6 @@ services:
     restart: always
     ports: 
       - 6000:6000/tcp
-      - 6200:6200/tcp
     volumes:
       - ../config/sharded/haproxy-shards.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
 
@@ -35,6 +34,10 @@ services:
     restart: on-failure
     environment:
       MareSynchronos__CdnFullUrl: "${DEV_MARE_CDNURL}"
+      MareSynchronos__CdnShardConfiguration__0__CdnFullUrl: "${DEV_MARE_CDNURL}"
+      MareSynchronos__CdnShardConfiguration__0__FileMatch: "^[01234567]"
+      MareSynchronos__CdnShardConfiguration__1__CdnFullUrl: "${DEV_MARE_CDNURL2}"
+      MareSynchronos__CdnShardConfiguration__1__FileMatch: "^[89ABCDEF]"
     volumes:
       - ../config/sharded/server-shard-main.json:/opt/MareSynchronosServer/appsettings.json
       - ../log/server-shard-main/:/opt/MareSynchronosServer/logs/:rw
@@ -98,6 +101,8 @@ services:
       - ../log/files-shard-1/:/opt/MareSynchronosStaticFilesServer/logs/:rw
       - postgres_socket:/var/run/postgresql/:rw
       - ../data/files-shard-1/:/marecache/:rw
+    ports: 
+      - 6200:6200/tcp
     depends_on:
       - "postgres"
       - "mare-files"
@@ -110,6 +115,8 @@ services:
       - ../log/files-shard-2/:/opt/MareSynchronosStaticFilesServer/logs/:rw
       - postgres_socket:/var/run/postgresql/:rw
       - ../data/files-shard-2/:/marecache/:rw
+    ports: 
+      - 6201:6200/tcp
     depends_on:
       - "postgres"
       - "mare-files"

Docker/run/config/sharded/haproxy-shards.cfg

@@ -23,17 +23,8 @@ frontend mare
         bind :6000
         default_backend mare-servers
 
-frontend mare-files
-        bind :6200
-        default_backend mare-files
-
 backend mare-servers
         balance leastconn
         cookie SERVER insert indirect nocache
         server mare1 mare-shard-1:6000 cookie mare1
         server mare2 mare-shard-2:6000 cookie mare2
-
-backend mare-files
-        balance roundrobin
-        server files1 mare-files-shard-1:6200
-        server files2 mare-files-shard-2:6200

Docker/run/config/sharded/server-shard-main.json

@@ -41,7 +41,17 @@
         "MaxJoinedGroupsByUser": 6,
         "MaxGroupUserCount": 100,
         "PurgeUnusedAccounts": false,
-        "PurgeUnusedAccountsPeriodInDays": 14
+        "PurgeUnusedAccountsPeriodInDays": 14,
+        "CdnShardConfiguration": [
+            {
+                "FileMatch": "^[01234567]",
+                "CdnFullUrl": ""
+            },
+            {
+                "FileMatch": "^[89ABCDEF]",
+                "CdnFullUrl": ""
+            }
+        ]
     },
     "AllowedHosts": "*",
     "Kestrel": {