Sending large files from server to client

[Timmy]

Context
I’m working on a module that displays ULX logs in XGUI.

Problem
• Active servers can have large log files (up to several megabytes).
• All data transfers in Garry’s Mod are throttled to 20 kb/s.
• A Net message can only contain up to 64 kb of data.

Current approach
Send compressed contents with xgui.SendDataEvent
+ Simple
- Fails when dealing with large log files

Alternative approaches
"Load more…" button
Send a snippet. Make user press button to load the next ~64 kb.
+ Fast, works with large files
- Clutters UI, inconvenient when dealing with large files

Send in chunks
Split contents into small chunks. Send all the chunks.
+ Works with large files
- Very slow and intense lag when dealing with large files

Remote server
Upload log to a remote server.
+ Access to all of the contents, fast
- Server can go offline, client may not have internet access, access to host may be blocked
Are there any other approaches I should consider? How would you send large files to users?

[JamminR]

Though still a type of "remote server", don't host remotely.
I just came up with this idea,and apparently other people have already made it happen.
Lua web server. (Does Gmod Awesomium limit file size?
Several exist.
https://www.google.com/search?q=lua+web+server
I wasn't overly surprised, as I use a Python CherryPy/Mako based web server, but, yeah, Lua.

[Timmy]

Thanks, JamminR! I appreciate your input.

Though none of those web servers are written in pure Lua. They all require binary modules written in C/C++. I don’t believe it is possible without those.

There is a networking module for Garry’s Mod written by a third party called Bromsock.

I expect this would be fast and work with large files.

But this would reduce accessibility as server owners would have to manually install (and update) the binary module. They would also have to update their firewall to allow traffic on an additional port.

I would also be relying on the maintainer’s ability to keep the project up-to-date and secure.

[JamminR]

Welcome. I didn't look that deep - figures - I'd hoped something would be available in pure lua.
Loading binary in this instance is cheating!

[MrPresident]

You could use an approach I used for sending screenshots from the client to the server. Should work backwards.

Take the log file and convert it to binary, then break it up into chunks and send a net message to the user with a header of the current chunk and the total chunks. Then have the client wait until it receives all the chunks and put them back together.

[Timmy]

Thanks for your input! I’m going to experiment with chunked transfers a bit more.

Garry’s Mod also performs a chunked transfer when sending save files to clients.

First few tests are… interesting. It turns out that you can only send 262112 bytes per tick. I am definitely going to have to limit the amount of data I send in each frame so I don’t overflow the buffer.

[Stickly Man!]

Although not very elegant, XGUI's data transfer system can be set up to send chunks of data, much like Mr. President is describing. I think I only used it for bandata because some servers had so many bans- but we've since changed the bans to be queried on demand for performance reasons.

[Timmy]

Oh, neat! I didn’t know that. I’ll be sure to take another look at the XGUI code related to data transfers. Thanks!

[Stickly Man!]

It's a pretty inefficient system- anything you come up with yourself would likely be much better. I've never written up a full guide on how the system works, so, here's a brief guide to get you started in the right direction. Some examples are pulled from the old bans code, which again, spent a lot of resources to ensure the players had ALL of the bans data, which ended up being too cumbersome. I ended up adding #6 to retrieve ban data on demand, but, as it bypasses all of the code that splits the data into chunks, it really just turned into a glorified ULib.clientRPC to send data through XGUI lol.

1) Register a datatype (serverside):
https://github.com/TeamUlysses/ulx/blob/6072e9d3ff5dc9ffeb65e9945b3ab6e275335d11/lua/ulx/xgui/server/sv_bans.lua#L8

Code: Lua

1. xgui.addDataType( "bans", function() return xgui.ulxbans end, "xgui_managebans", 30, 20 )

Parameters are:
 - name of datatype (must be unique)
 - function that will return all of the data to be sent to players
 - access tag the client must have access to to get this data
 - maximum number of items to send in a "chunk" (0 to send all data). Note that this just limits by entries in the top-level table, so it's pretty impossible to limit an exact size in bytes.
 - priority level of the data (-20 to 20, higher number means lower priority. Used to determine what order to send the datatypes in). Was useful when XGUI could potentially be downloading a lot of data, which is no longer the case.

With this, XGUI will ensure the player has access to the data they need, especially after their access level changes. However, this will ensure they always have ALL of the data (bad if you're working with a lot of data).

2) When data is changed on the server and needs to be re-sent to the client, you can manually resend all of the data (serverside). This is the easiest but least efficient option:
https://github.com/TeamUlysses/ulx/blob/master/lua/ulx/xgui/server/sv_settings.lua#L38

Code: Lua

1. xgui.sendDataTable( {}, "gimps" )

Parameters:
 - List of players who should receive the data. Empty table to send to all. XGUI will always filter out users who don't have access to the data.
 - One or more names of dataTypes of the data you want to send. String or table allowed.

3) (Optional) For more efficient data syncing, you have the ability to add, remove, or update a piece of data (serverside).
https://github.com/TeamUlysses/ulx/blob/6072e9d3ff5dc9ffeb65e9945b3ab6e275335d11/lua/ulx/xgui/server/sv_bans.lua#L119

Code: Lua

1. xgui.addData( {}, "bans", t )

https://github.com/TeamUlysses/ulx/blob/6072e9d3ff5dc9ffeb65e9945b3ab6e275335d11/lua/ulx/xgui/server/sv_bans.lua#L130

Code: Lua

1. xgui.removeData( {}, "bans", { steamid } )

https://github.com/TeamUlysses/ulx/blob/master/lua/ulx/xgui/server/sv_groups.lua#L50

Code: Lua

1. xgui.updateData( {}, "users", temp )

Parameters are (See #2 for more details):
 - List of players
 - Name of datatype (string only)
 - Data to be inserted, keys to be removed, or data to be replaced.

XGUI will automatically update the client's data tables to match the changes. For more details, see the linked code examples above, or see how the client actually pieces the data back into the table, and check these comments for some kind of documentation.

4) Prepare XGUI to be aware of your data type: (clientside)
https://github.com/TeamUlysses/ulx/blob/6072e9d3ff5dc9ffeb65e9945b3ab6e275335d11/lua/ulx/xgui/bans.lua#L4

Code: Lua

1. xgui.prepareDataType( "bans" )

Your data can now be accessed in the 'xgui.data.<datatype>' table.

5) (Optional) Hook into XGUI data events to be notified when your data updates: (clientside)
https://github.com/TeamUlysses/ulx/blob/6072e9d3ff5dc9ffeb65e9945b3ab6e275335d11/lua/ulx/xgui/bans.lua#L488-L491

Code: Lua

1. xgui.hookEvent( "bans", "process", xbans.populateBans )
2. xgui.hookEvent( "bans", "clear", xbans.clearbans )
3. xgui.hookEvent( "bans", "add", xbans.banUpdated )
4. xgui.hookEvent( "bans", "remove", xbans.banRemoved )

These methods should be passed a single parameter with the same data sent by the data parameter in #3, in case you need to dig into it. The full list of data events you can hook into can be found here, and you can check here to see exactly where they are called.

6) Advanced: Send your own set of data. This is what I'm currently using for bans that will, when requested by the user, send a page of bans to be displayed in their UI. Unfortunately, this bypasses the splitting data into chunks. I should probably fix that at some point lol.
https://github.com/TeamUlysses/ulx/blob/master/lua/ulx/xgui/server/sv_bans.lua#L237

Code: Lua

1. xgui.sendDataEvent( ply, 7, "bans", bansToSend )

Here's where the clientside code listens for and handles this event. XGUI will not automatically do anything with this data:
https://github.com/TeamUlysses/ulx/blob/master/lua/ulx/xgui/bans.lua#L446-L451

Code: Lua

1. function xbans.banPageRecieved( data )
2.         xgui.data.bans.cache = data
3.         xbans.clearbans()
4.         xbans.populateBans()
5. end
6. xgui.hookEvent( "bans", "data", xbans.banPageRecieved, "bansGotPage" )

.. That ended up being a more thorough rundown than I intended to make. Oh well, it's a good refresher for me since I've forgotten how most of it is pieced together. Hope you find it somewhat useful!

[Timmy]

Wow, terrific!

I really appreciate that you took the time to document the relevant functions. Your post is going to be an excellent reference whenever I use the XGUI data transfer system. :)