Complete Scan of Dynamodb with Boto3

Scan DynamoDB with boto3

I was able to successfully solve this with the FilterExpression as suggested in a previous comment.

Here is what I came up with:

    import boto3
    from boto3.dynamodb.conditions import Key

    fe = Key('attribute').eq('value')

    response = table.scan(
    FilterExpression=fe)
    data = response['Items']

With boto3 Dynamodb scan is there a way to get changes of events rather than last update

Is there a way to get the previous two changes as well?

Sadly, there is not unless you implement such a solution for that yourself. Probably you would have to use DynamoDB Streams to stream changes to a lambda function, and then store all historical values in some other database.

Scan Dynamo DB with boto3 for array of dictionary

@tommy, Your answer is almost correct except we should not use json.dumps

itemVal = {"address": "aaa@gmail.com", "type": "email"}

response = table.scan(
    FilterExpression=Attr('toAddr').contains((itemVal)
) 

The above one helped to scan results.

full table scanning using boto3 python

The key question is how many items available on your Dynamodb table?

I will just explain the above code especially the while loop that you have added lately.

The while loop is to scan all the items on DynamoDB table until there is no items to scan. One table scan will bring you only 1 MB of data. So, it has to be executed recursively while all the items are scanned.

while 'LastEvaluatedKey' in response:

If the total number of scanned items exceeds the maximum data set size
limit of 1 MB, the scan stops and results are returned to the user as
a LastEvaluatedKey value to continue the scan in a subsequent
operation.

Imagine, if you have millions of items in the table, the program has to scan all the items recursively before it ends. Also, the filter criteria is applied on the scan result set.

Due to the reasons mentioned above, mostly the table scan should be avoided on big tables. The scan process is normally inefficient and it would cost you as well.

Alternate solution - Use Global Secondary Index (GSI) and Query API

---

Related Topics