Ideas needed

Discussion in 'Software' started by zxspectrum, Jan 27, 2022.

  1. zxspectrum

    zxspectrum Terabyte Poster Forum Leader Gold Member

    2,078
    212
    244
    Hello all

    Been thinking of an easy way to search a large volume of logs for which we get by the hour. The logs contain vast amounts of information and in this particular case, I would like to start to search for someone who has deleted something. When we get these sorts of requests in one thing we don't get is a set date, the date we do get is often between the space of 5 days and depending onthe time of the day, the logs can be 50 MB in size - for 1 hour, which roughly is about 6 b of data given a 5 day period, its also all text so there is a lot to sift through

    One thing I forgot to mention is that easy hour is also split into three text files. After some testing, it does appear you can search form things, I set up a dummy test and then deleted the test and not all the data refers to a user directly, what I mean by that, if I was searching for Barry, this would come up as an external key like 12345 for instance.

    The long and short of it is would anyone have an idea of how I would progress this, what I mean is that, I would like to download the data for a given day or period - put it somewhere - possibly a data base (which I would have to create I think and have no idea where to start

    Would anyone have any pointers at all?

    Cheers
    Ed
     
    Certifications: BSc computing and information systems
    WIP: 70-680
  2. dmarsh
    Honorary Member

    dmarsh Petabyte Poster

    4,270
    481
    219
    There are lots of realtime log aggregators out there, Kibana / Logstash, etc.

    Simpler tasks you can use grep or write your own program.

    What databases give you is indexes, query plans and caching.

    Larger scale stuff people use Kafka, BigQuery or Apache Spark.
     
  3. Nyx

    Nyx Byte Poster

    188
    22
    15
    You can do this with powershell, not sure how long it would take though.

    loop to go through each file in a folder (assuming you have all files in one)
    get content of the file
    loop to go through each line of the file
    match keyword - print the line out

    literally few lines of code
     
  4. zxspectrum

    zxspectrum Terabyte Poster Forum Leader Gold Member

    2,078
    212
    244
    Cheers for the replies - another rabbit hole to go down ha ha

    Ed
     
    Certifications: BSc computing and information systems
    WIP: 70-680
  5. Nyx

    Nyx Byte Poster

    188
    22
    15
    at its simplest something like this:

    $folder = "c:\" ## path to files
    $files = get-childitem -Path $folder
    $searchterm = "" ## what to search for
    foreach ($item in $files){
    $file = get-content $item.FullName
    foreach ($line in $file){
    if ($line -match $searchterm) {$line}
    }
    }

    maybe test it on a small batch first.
     
  6. zxspectrum

    zxspectrum Terabyte Poster Forum Leader Gold Member

    2,078
    212
    244
    Cheers Nyx
    Will give that a go

    Ed
     
    Certifications: BSc computing and information systems
    WIP: 70-680
  7. AdamGreen22

    AdamGreen22 New Member

    4
    0
    1
    Thanks a lot for help!
     

Share This Page

Loading...
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.