How to Use Ruby CSV Converters

How do you use Ruby CSV converters?

Your date times don't match the CSV::DateTimeMatcher regexp that CSV uses to decide whether it should attempt a date time conversion. By the looks of it it's doing so because of the fractional seconds you've got.

You could either overwrite that constant or write your own converter (DateTime.parse seems happy with your strings)

How to add a custom convert to ruby csv?

You can pass the same options to parse that you can pass to new, and with new you can pass in an array of converters, which can be the name of one of the built-in converters, or a lambda:

table = CSV.parse(csv_data, headers: true, converters: [
  :integer,
  -> field, info { 'Credit' == info.header && field.empty? ? 0 : field },
  -> field, info { 'Debit' == info.header && field.empty? ? 0 : field },
])

p table # => #<CSV::Table mode:col_or_row row_count:2>
puts table[0]['Debit'] # => 10
puts table[0]['Credit'] # => 0

Using Ruby CSV header converters

After doing some research here in my desktop, it seems to me the error is for something else.

First I put the data in my "a.txt" file as below :

First Name,Last Name
John,Doe
Jane,Doe

Now I ran the code, which is saved in my so.rb file.

so.rb

require 'csv'

CSV.foreach("C:\\Users\\arup\\a.txt", 
             :headers => true,
             :converters => :all,
             :header_converters => lambda { |h| h.downcase.gsub(' ', '_') }
           ) do |row|
    p row
end

Now running the :

C:\Users\arup>ruby -v so.rb
ruby 1.9.3p448 (2013-06-27) [i386-mingw32]
#<CSV::Row "first_name":"John" "last_name":"Doe">
#<CSV::Row "first_name":"Jane" "last_name":"Doe">

So everything is working now. Now let me reproduce the error :

I put the data in my "a.txt" file as below ( just added a , after the last column) :

First Name,Last Name,
John,Doe
Jane,Doe

Now I ran the code, which is saved in my so.rb file, again.

C:\Users\arup>ruby -v so.rb
ruby 1.9.3p448 (2013-06-27) [i386-mingw32]
so.rb:5:in `block in <main>': undefined method `downcase' for nil:NilClass (NoMethodError)

It seems, in your header row, there is blank column value which is causing the error. Thus if you have a control to the source CSV file, check there the same. Or do some change in your code, to handle the error as below :

require 'csv'

CSV.foreach("C:\\Users\\arup\\a.txt",
             :headers => true,
             :converters => :all,
             :header_converters => lambda { |h| h.downcase.gsub(' ', '_') unless h.nil? }
           ) do |row|
    p row
end

Custom Ruby CSV converters for multiple fields

This is the proper way to do things if the default converters aren't sufficient. My only suggestion would be to separate your converters into different lambdas since the CSV library is already designed to test each field against an array of converters (making your case redundant).

But if this is just a quick one-off script, what you have is good enough.

CSV.generate and converters?

You need to write your converter as as below :

CSV::Converters[:nonewline] = lambda do |s|
  s.gsub(/(\r?\n)+/,' ')
end

Then do :

CSV.generate(:converters => [:nonewline]) do |csv|
  csv << ["hello\ngoodbye"]
end

Read the documentation Converters .

Okay, above part I didn't remove, as to show you how to write the custom CSV converters. The way you wrote it is incorrect.

Read the documentation of CSV::generate

This method wraps a String you provide, or an empty default String, in a CSV object which is passed to the provided block. You can use the block to append CSV rows to the String and when the block exits, the final String will be returned.

After reading the docs, it is quite clear that this method is for writing to a csv file, not for reading. Now all the converters options ( like :converters, :header_converters) is applied, when you are reading a CSV file, but not applied when you are writing into a CSV file.

Let me show you 2 examples to illustrate this more clearly.

require 'csv'

string = <<_
foo,bar
baz,quack
_

File.write('a',string)

CSV::Converters[:upcase] = lambda do |s|
  s.upcase
end

I am reading from a CSV file, so :converters option is applied to it.

CSV.open('a','r',:converters => :upcase) do |csv|
  puts csv.read
end

output

# >> FOO
# >> BAR
# >> BAZ
# >> QUACK

Now I am writing into the CSV file, converters option is not applied.

CSV.open('a','w',:converters => :upcase) do |csv|
  csv << ['dog','cat']
end

CSV.read('a') # => [["dog", "cat"]]

Ruby CSV converter, remove all converters?

Are you sure that your CSV file doesn't actually contain the 4-digit year? Try looking at puts File.read('original-data.csv')

When I tried this on Ruby 2.1.8, it didn't change the value

require 'csv'

my_csv_data = 'hello,"9/30/14 0:00",world'

CSV.new(my_csv_data).each do |row|
  puts row.inspect # prints ["hello", "9/30/14 0:00", "world"], as expected
end

Fetch the data from the CSV files in ruby and Order the Data in DESC?

You need column names in the CSV if you want to use the row["COL"] csv syntax.
Additionally you need to use += not = to add them up. Right now you're just overwriting the value every time.

Name,Sales
Randy, 200
Robin, 502
Randy, 200
Raj,   502
Randy, 500      
Robin, 102      
Mano,  220
Raj,   502
Randy, 285
Robin, 385
Randy, 295
Raj,   596

require 'csv'
cust = Hash.new

CSV.foreach(("./data.csv"), headers: true, col_sep: ",") do |row|
  # if key isn't in hash start it at 0
  unless cust.keys.include?(row["Name"])
    cust[row["Name"]] = 0 
  end
  # add the sales (.to_i converts to integer)
  cust[row["Name"]] += row["Sales"].to_i
end

puts "Name Sales Rank"
# Sort by the value not the key. Make it negative so it's descending
# not ascending. 
# each_with_index is just a nice way to count them
cust.sort_by{|k,v| -v}.each_with_index do |(name, sales), i|
  puts "#{name} #{sales} #{i+1}"
end

Produces:

Raj 1600 1
Randy 1480 2
Robin 989 3
Mano 220 4

How to observe a stream in Ruby's CSV module?

I can recommend one of two approaches, depending on your needs and personal taste.

I have intentionally distilled the code to just its bare minimum (without your wrapping class), for clarity.

1. Simple read-modify-write loop

Since you do not want to slurp the file, use CSV::Foreach. For example, for a quick debugging session, do:

CSV.foreach "source.csv", headers: true do |row|
  row["name"] = row["name"].upcase
  row["new column"] = "new value"
  p row
end

And if you wish to write to file during that same iteration:

require 'csv'

csv_options = { headers: true }

# Open the target file for writing
CSV.open("target.csv", "wb") do |target|
  # Add a header
  target << %w[new header column names]

  # Iterate over the source CSV rows
  CSV.foreach "source.csv", **csv_options do |row|
    # Mutate and add columns
    row["name"] = row["name"].upcase
    row["new column"] = "new value"

    # Push the new row to the target file
    target << row
  end
end

2. Using CSV::Converters

There is a built in functionality that might be helpful - CSV::Converters - (see the :converters definition in the CSV::New documentation)

require 'csv'

# Register a converter in the options hash
csv_options = { headers: true, converters: [:stripper] }

# Define a converter
CSV::Converters[:stripper] = lambda do |value, field|
  value ? value.to_s.strip : value
end

CSV.open("target.csv", "wb") do |target|
  # same as above

  CSV.foreach "source.csv", **csv_options do |row|
    # same as above - input data will already be converted
    # you can do additional things here if needed
  end
end

3. Separate input and output from your converter classes

Based on your comment, and since you want to minimize I/O and iterations, perhaps extracting the read/write operations from the responsibility of the transformers might be of interest. Something like this.

require 'csv'

class NameCapitalizer
  def self.call(row)
    row["name"] = row["name"].upcase
  end
end

class EmailRemover
  def self.call(row)
    row.delete 'email'
  end
end

csv_options = { headers: true }
converters = [NameCapitalizer, EmailRemover]

CSV.open("target.csv", "wb") do |target|
  CSV.foreach "source.csv", **csv_options do |row|
    converters.each { |c| c.call row }
    target << row
  end
end

Note that the above code still does not handle the header, in case it was changed. You will probably have to reserve the last row (after all transformations) and prepend its #headers to the output CSV.

There are probably plenty other ways to do it, but the CSV class in Ruby does not have the cleanest interface, so I try to keep code that deals with it as simple as I can.

---

Related Topics