Ideas needed

Oh man! Ad-blocking software has been detected! :'(

This website is run by the community, for the community... and it needs advertisements in order to keep running. Blocking our ads means your killing our stats!
Please disable your ad-block, or become a premium member to hide all advertisements and this notice.

Hello all

Been thinking of an easy way to search a large volume of logs for which we get by the hour. The logs contain vast amounts of information and in this particular case, I would like to start to search for someone who has deleted something. When we get these sorts of requests in one thing we don't get is a set date, the date we do get is often between the space of 5 days and depending onthe time of the day, the logs can be 50 MB in size - for 1 hour, which roughly is about 6 b of data given a 5 day period, its also all text so there is a lot to sift through

One thing I forgot to mention is that easy hour is also split into three text files. After some testing, it does appear you can search form things, I set up a dummy test and then deleted the test and not all the data refers to a user directly, what I mean by that, if I was searching for Barry, this would come up as an external key like 12345 for instance.

The long and short of it is would anyone have an idea of how I would progress this, what I mean is that, I would like to download the data for a given day or period - put it somewhere - possibly a data base (which I would have to create I think and have no idea where to start

Would anyone have any pointers at all?

Cheers
Ed

 

Oh man! Ad-blocking software has been detected! :'(

This website is run by the community, for the community... and it needs advertisements in order to keep running. Blocking our ads means your killing our stats!
Please disable your ad-block, or become a premium member to hide all advertisements and this notice.

There are lots of realtime log aggregators out there, Kibana / Logstash, etc.

Simpler tasks you can use grep or write your own program.

What databases give you is indexes, query plans and caching.

Larger scale stuff people use Kafka, BigQuery or Apache Spark.

 

You can do this with powershell, not sure how long it would take though.

loop to go through each file in a folder (assuming you have all files in one)
get content of the file
loop to go through each line of the file
match keyword - print the line out

literally few lines of code

 

Cheers for the replies - another rabbit hole to go down ha ha

Ed

 

at its simplest something like this:

$folder = "c:\" ## path to files
$files = get-childitem -Path $folder
$searchterm = "" ## what to search for
foreach ($item in $files){
$file = get-content $item.FullName
foreach ($line in $file){
if ($line -match $searchterm) {$line}
}
}

maybe test it on a small batch first.

 

Cheers Nyx
Will give that a go

Ed

 

Thanks a lot for help!

 

You were very helpful, thank you!