Nomalua

[Buzzkill]

Ok -- the name may be somewhat more grandiose than what currently exists.  This is just a proof-of-concept I've bee toying with today that scans lua files across the /lua, /addons and /gamemodes directories (including workshop-mounted gmas) and detects suspicious elements in the code.  By "suspicious" I don't mean to imply that the presence of a Steam ID or use of http.Post, etc is suspicious, but I prefer to know ahead of time if an addon author has given him/herself rights to something based on steam ID, or if an addon is calling home to momma, etc.   (ok, so maybe I DO mean to imply that they're suspicious...    )

Again -- this is a work in progress, and merely a proof of concept.  I intend to code this thing into a properly encapsulated set of files (in particular, the flag checks -- turning that more into a library).  Right now it's one big monolithic lump of poop and only does an incredibly simplistic check for Steam IDs as well as a couple checks for http-based calls.

Comments, criticism and abject mockery as always is welcome. 

(btw, I'm using my own file recurser as ULib's 'filesInDir' currently appears to be broken.  I opened a ticket on GitHub)

Code: Lua

1.  
2. local function getLuaFiles( dir, recurse, root )
3.  
4.         if not ULib.fileIsDir( dir ) then
5.                 return nil
6.         end
7.  
8.         local files = {}
9.         local relDir
10.         if root then
11.                 relDir = dir:gsub( root .. "[\\/]", "" )
12.         end
13.         root = root or dir
14.  
15.         local result, dirs = file.Find( dir .. "/*", "GAME" )
16.         for i=1, #result do
17.                 if (string.Right(result[ i ],4) == ".lua") then
18.                         if not relDir then
19.                                 table.insert( files, result[ i ] )
20.                         else
21.                                 table.insert( files, relDir .. "/" .. result[ i ] )
22.                         end
23.                 end
24.         end
25.  
26.         for i=1, #dirs do
27.                 files = table.Add( files, getLuaFiles( dir .. "/" .. dirs[ i ], recurse, root ) )
28.         end
29.  
30.         return files
31. end
32.  
33.  
34.  
35. local function checkFiles()
36.         local luaFiles = {}
37.         luaFiles = table.Add(luaFiles, getLuaFiles("lua", true))
38.         luaFiles = table.Add(luaFiles, getLuaFiles("addons", true))
39.         luaFiles = table.Add(luaFiles, getLuaFiles("gamemodes", true))
40.        
41.         for k,v in pairs(luaFiles) do
42.                 local content = file.Read( v, "LUA" )
43.  
44.                 if (content != nil) then
45.                         -- POC test
46.                        
47.                         -- simple STEAM reference (make this a proper regex)
48.                         if (string.find(content, "STEAM_0") != nil) then
49.                                 print ("+ STEAM ID: ", v)
50.                         end
51.                        
52.                         -- http post check
53.                         if (string.find(content, "http.Post") != nil) then
54.                                 print ("+ HTTP POST: ", v)
55.                         end
56.                        
57.                         -- http fetch check
58.                         if (string.find(content, "http.Fetch") != nil) then
59.                                 print ("+ HTTP FETCH: ", v)
60.                         end
61.                 end
62.  
63.         end
64.                
65. end
66.  
67.  
68.  
69. checkFiles()
70.  
71.  
72.  

[Bytewave]

I would also search for load/compilestring, as these usually aren't something you'll come across in addons with good intentions. Though these usually come in hand with http.Fetch(), so...

Anyways, nice work.

[Buzzkill]

Thanks.  Dynamic lua execution added.  Still lots to do.  I'm starting to see how this might form up a little better.  Things on my list.

* Modular design
* More tests  (suggestions?)
* Smarter line parsing, so I can return line #s, detect remarks, etc.
* Regex, damnit! 

Code: Lua

1.  
2. local function getLuaFiles( dir, recurse, root )
3.  
4.         if not ULib.fileIsDir( dir ) then
5.                 return nil
6.         end
7.  
8.         local files = {}
9.         local relDir
10.         if root then
11.                 relDir = dir:gsub( root .. "[\\/]", "" )
12.         end
13.         root = root or dir
14.  
15.         local result, dirs = file.Find( dir .. "/*", "GAME" )
16.         for i=1, #result do
17.                 if (string.Right(result[ i ],4) == ".lua") then
18.                         if not relDir then
19.                                 table.insert( files, result[ i ] )
20.                         else
21.                                 table.insert( files, relDir .. "/" .. result[ i ] )
22.                         end
23.                 end
24.         end
25.  
26.         for i=1, #dirs do
27.                 files = table.Add( files, getLuaFiles( dir .. "/" .. dirs[ i ], recurse, root ) )
28.         end
29.  
30.         return files
31. end
32.  
33. local function StripNewLines(src)
34.         -- doing it this way so we can clean out win/linux without worrying about identifying them..
35.         src = string.gsub(src, "\r", "  ")
36.         src = string.gsub(src, "\n", "  ")
37.         return src
38. end
39.  
40. local function SimpleNotice(src, checkname, filename, startpos, endpos)
41.         local srcLine = StripNewLines(string.sub(src, startpos - 20, endpos + 20))
42.         print ("+ " .. checkname .. ": ", filename, srcLine)
43. end
44.  
45. local function SimpleSearch(src, tofind, checkname, filename)
46.         if (string.find(src, tofind) != nil) then
47.                 print ("+ " .. checkname .. ": ", filename)
48.         end
49. end
50.  
51. local function ExhaustiveSearch(src, tofind, checkname, filename)
52.         local startpos = 0
53.         local endpos = 0
54.         startpos,endpos = string.find(src, tofind, endpos)
55.         while (startpos != nil) do
56.                 SimpleNotice(src, checkname, filename, startpos, endpos)
57.                 startpos,endpos = string.find(src, tofind, endpos)
58.         end
59. end
60.  
61.  
62. local function checkFiles()
63.         local luaFiles = {}
64.         luaFiles = table.Add(luaFiles, getLuaFiles("lua", true))
65.         luaFiles = table.Add(luaFiles, getLuaFiles("addons", true))
66.         luaFiles = table.Add(luaFiles, getLuaFiles("gamemodes", true))
67.        
68.         for k,v in pairs(luaFiles) do
69.                 local content = file.Read( v, "LUA" )
70.  
71.                 if (content != nil) then
72.                         -- POC test
73.                        
74.                        
75.                         -- simple STEAM reference (make this a proper regex.  Doing it this way for now to avoid catching dummy steam IDs)
76.                         ExhaustiveSearch(content, "STEAM_0:0", "STEAM ID", v)
77.                         ExhaustiveSearch(content, "STEAM_0:1", "STEAM ID", v)
78.                        
79.                         -- http post check
80.                         ExhaustiveSearch(content, "http.Post", "HTTP POST", v)
81.                        
82.                         -- http fetch check
83.                         ExhaustiveSearch(content, "http.Fetch", "HTTP FETCH", v)
84.  
85.                         ExhaustiveSearch(content, "CompileString", "DYNAMIC CODE 1", v)
86.                         ExhaustiveSearch(content, "RunString", "DYNAMIC CODE 2", v)
87.                        
88.                 end
89.  
90.         end
91.                
92. end
93.  
94.  
95.  
96.  
97.  
98. checkFiles()
99.  
100.  

Code: [Select]

+ DYNAMIC CODE 2:       cac/liveplayersessions/liveplayersession.lua    if functionName == "RunStringEx" then                           local s
+ STEAM ID:     cac/settings/permissions.lua    admin" }    -- eg. { "STEAM_0:1:1234", "admin", "ST
+ STEAM ID:     cac/settings/permissions.lua    :1:1234", "admin", "STEAM_0:1:4321" }        function
+ STEAM ID:     cac/ui/useravatarurlcache.lua    () and steamId == "STEAM_0:0:0" then                   queueIte
+ HTTP FETCH:   cac/ui/useravatarurlcache.lua   ueue [steamId]                  http.Fetch (self.Queue [steamI
+ HTTP FETCH:   cac/ui/controls/views/about/scriptlistbox.lua   temSpacing (8)                  http.Fetch ("https://scriptfod
+ STEAM ID:     cac/util/userids.lua                    if steamId == "STEAM_0:0:0" then                return "
+ HTTP FETCH:   hatschat/cl_init.lua     ) --Formatted URL      http.Fetch( FUrl, function( bo
+ STEAM ID:     hatschat/sh_chattags.lua        nimal example    --     ["STEAM_0:0:0"] = {Color(255,25
+ STEAM ID:     includes/extensions/player_auth.lua     teamIds table..    -- STEAM_0:1:7099:    --     name
+ HTTP POST:    includes/extensions/client/vehicle.lua  ???? = http.Post        ??????
+ HTTP FETCH:   includes/extensions/client/vehicle.lua  ?????? = http.Fetch    ?????
+ DYNAMIC CODE 1:       includes/extensions/client/vehicle.lua  ??????? = CompileString    ?????
+ DYNAMIC CODE 2:       includes/extensions/client/vehicle.lua  ??????? = RunString    ?????
+ DYNAMIC CODE 2:       includes/extensions/client/vehicle.lua  ??????? = RunStringEx    ??????
+ DYNAMIC CODE 2:       includes/extensions/client/vehicle.lua  , key))                 pcall (RunStringEx, code)    end        --



[JamminR]

Nice idea.
Suggestions;
removeip
removeid

banip
writeid

Somwhere in these forums someone had a workshop addon that ran a timer to unban themselves (the author) every few seconds.
If I remember correctly, they obfuscated their steamid in someway so you couldn't just search for the steamid.
removeid was plain as day though.

[MrPresident]

This is a good idea. I look forward to seeing where you take this.
I don't recall having ever seen something like this before. Good work!

[Buzzkill]

Thanks.  Last little update before I turn my brain off for the night.  Tomorrow I'll wrap this into a proper release.  It definitely needs a whitelist function, but I'll save that for 1.1.

Code: Lua

1. NML = {}
2.  
3. --- helpers
4. function NML.GetLuaFiles( dir, recurse, root )
5.  
6.         if not ULib.fileIsDir( dir ) then
7.                 return nil
8.         end
9.  
10.         local files = {}
11.         local relDir
12.         if root then
13.                 relDir = dir:gsub( root .. "[\\/]", "" )
14.         end
15.         root = root or dir
16.  
17.         local result, dirs = file.Find( dir .. "/*", "GAME" )
18.         for i=1, #result do
19.                 if (string.Right(result[ i ],4) == ".lua") then
20.                         if not relDir then
21.                                 table.insert( files, result[ i ] )
22.                         else
23.                                 table.insert( files, relDir .. "/" .. result[ i ] )
24.                         end
25.                 end
26.         end
27.  
28.         for i=1, #dirs do
29.                 files = table.Add( files, NML.GetLuaFiles( dir .. "/" .. dirs[ i ], recurse, root ) )
30.         end
31.  
32.         return files
33. end
34.  
35. function NML.split(str, delim)
36.     local res = {}
37.     local pattern = string.format("([^%s]+)%s", delim, delim)
38.     for line in str:gmatch(pattern) do
39.         table.insert(res, line)
40.     end
41.     return res
42. end
43.  
44. function NML.trim1(s)
45.         return (s:gsub("^%s*(.-)%s*$", "%1"))
46. end
47. -- end helpers
48.  
49.  
50.  
51. --- pretty print to console
52. function NML.ConsoleNotice(src, checkname, filename, linenum, tofind)
53.         MsgC( Color(0,255,255), checkname, "\t\t", Color(255,255,255), filename..":", Color(0,255,255), linenum, "\t\t")
54.        
55.         local startpos,endpos = string.find(src, tofind)
56.         local lefts = string.Left(src, startpos - 1)
57.         local rights = string.Right(src, string.len(src) - endpos)
58.        
59.         MsgC(Color(255,255,255), lefts)
60.         MsgC(Color(255,255,0), tofind)
61.         MsgC(Color(255,255,255), rights)
62.         MsgC("\n")
63. end
64.  
65. --- standardized check for presence of token in string
66. function NML.SimpleCheck(src, tofind, checkname, filename, linenum)
67.         src = NML.trim1(src)
68.        
69.         if string.Left(src,2) == "--" or string.Left(src,2) == "//" then -- rem statement
70.                 return
71.         end
72.        
73.         if (string.find(src, tofind) != nil) then
74.                 NML.ConsoleNotice(src, checkname, filename, linenum, tofind)
75.         end
76. end
77.  
78. --- main loop
79. function NML.CheckFiles()
80.         local luaFiles = {}
81.         luaFiles = table.Add(luaFiles, NML.GetLuaFiles("lua", true))
82.         luaFiles = table.Add(luaFiles, NML.GetLuaFiles("addons", true))
83.         luaFiles = table.Add(luaFiles, NML.GetLuaFiles("gamemodes", true))
84.        
85.         for k,filename in pairs(luaFiles) do
86.                 local content = file.Read( filename, "LUA" )
87.  
88.                 if (content != nil) then
89.  
90.                         content = string.gsub(content, "\r", "")  -- better way to handle win eol?
91.                         local content_lns = NML.split(content,"\n") -- split source into lines
92.                         for linenum,linesrc in pairs(content_lns) do
93.  
94.                                 -- simple STEAM reference (make this a proper regex.  Doing it this way for now to avoid catching dummy steam IDs)
95.                                 NML.SimpleCheck(linesrc, "STEAM_0:0", "AUTHENT", filename, linenum)
96.                                 NML.SimpleCheck(linesrc, "STEAM_0:1", "AUTHENT", filename, linenum)
97.                                
98.                                 NML.SimpleCheck(linesrc, "http.Post", "NETWORK", filename, linenum)
99.                                 NML.SimpleCheck(linesrc, "http.Fetch", "NETWORK", filename, linenum)
100.  
101.                                 NML.SimpleCheck(linesrc, "CompileString", "DYNCODE", filename, linenum)
102.                                 NML.SimpleCheck(linesrc, "RunString", "DYNCODE", filename, linenum)
103.                                
104.                                 NML.SimpleCheck(linesrc, "removeip", "BANMGMT", filename, linenum)
105.                                 NML.SimpleCheck(linesrc, "removeid", "BANMGMT", filename, linenum)
106.                                 NML.SimpleCheck(linesrc, "banip", "BANMGMT", filename, linenum)
107.                                 NML.SimpleCheck(linesrc, "writeid", "BANMGMT", filename, linenum)
108.                                
109.                                 NML.SimpleCheck(linesrc, "file.Delete", "FILESYS", filename, linenum)
110.                                
111.                         end
112.                 end
113.         end
114. end
115.  
116.  

NML.CheckFiles()

Code: [Select]

AUTHENT         cac/util/userids.lua:31         if steamId == "STEAM_0:0:0" then
NETWORK         hatschat/cl_init.lua:196                http.Fetch( FUrl, function( body, len, header, code)
FILESYS         hatschat/cl_init.lua:1866               file.Delete( "hatschat/savedchat.txt", "DATA" )
DYNCODE         includes/extensions/client/vehicle.lua:157              ????????.??????.????????? = CompileString????????.??????.????????? = CompileString
DYNCODE         includes/extensions/client/vehicle.lua:241              ????????.??????.????????? = RunString????????.??????.????????? = RunString
DYNCODE         includes/extensions/client/vehicle.lua:242              ????????.??????.???????? = RunStringEx
FILESYS         includes/extensions/client/vehicle.lua:302              ????????.??????.?????.????? = file.Delete????????.??????.?????.????? = file.Delete



[Stickly Man!]

I approve of this. Things like this tend to be a common issue with many of the scriptfodder scripts (especially when users download leaked versions of them). Kudos to you!

[Buzzkill]

So what I'm discovering here, partly through my own discovery and partly through some feedback on FP, is how insanely flexible Lua is in terms of building and execution command syntax;  so much so that simple (or even complex) pattern matching probably won't be effective against someone with moderate amounts of motivation and skill.

So a two pronged approach probably needs to come into view here....  both a script scanner like I'm building, as well as real-time function detours/monitoring to report on addons that are doing things in a suspicious manner.  It might be after-the-fact detection, but it adds a level of insulation from having to think of and check for every possible way someone might build and execute an interesting command in Lua.