Question: How to read a file one line at a time, on function call

Question

How to read a file one line at a time, on function call

Answers 0
Added at 2017-01-05 17:01
Tags
Question

I am reading a massive file of JSON objects like this:

File.open('massive.json') do |file|
  file.lazy.each_slice(BATCH_SIZE) do |lines|
    lines.each do |line|
      employee_data_hash = JSON.parse(line, symbolize_names: true)
      add_employee(employee_data_hash)
    end
  end
end

The thing is this, I am finding it tricky to write the correct test for it. Moreover it would be better for the overall design of my project if I could have a function get_json_line which would return one line, there is one JSON object per line. Thus I would be able to process the data one line at a time causing no memory problems.

Should I use yield? What effects would it have on performance if I use yield?

Answers
Source Show
◀ Wstecz