1. 遍历整个列表
# removing duplicated from list # using naive methods # initializing listtest_list = [1, 3, 5, 6, 3, 5, 6, 1]print ("The original list is : " + str(test_list))# using naive method to remove duplicated from list res = []for i in test_list:if i not in res:res.append(i)# printing list after removal print ("The list after removing duplicates : " + str(res))# 输出结果：# 原始列表是：[1, 3, 5, 6, 3, 5, 6, 1]# 删除重复项后的列表：[1, 3, 5, 6]
2.使用 set()
最大的缺点之一是set后列表中元素的顺序不再和原来一样
# removing duplicated from list # using set()# initializing listtest_list = [1, 5, 3, 6, 3, 5, 6, 1]print ("The original list is : " + str(test_list))# using set()to remove duplicated from list test_list = list(set(test_list))# printing list after removal # distorted orderingprint ("The list after removing duplicates : " + str(test_list))# 输出结果：# 原始列表是：[1, 5, 3, 6, 3, 5, 6, 1]# 删除重复项后的列表：[1, 3, 5, 6]
3. 使用列表理解 + enumerate()
此方法使用枚举根据列表理解删除重复元素。通过检查该元素是否已存在于列表中来跳过该元素。此方法保持列表中元素的顺序。
# removing duplicated from list # using list comprehension + enumerate()# initializing listtest_list = [1, 5, 3, 6, 3, 5, 6, 1]print ("The original list is : " + str(test_list))# using list comprehension + enumerate()# to remove duplicated from list res = [i for n, i in enumerate(test_list) if i not in test_list[:n]]# printing list after removal print ("The list after removing duplicates : " + str(res))
4.使用 collections.OrderedDict.fromkeys()
这是完成特殊任务的最快方式,它首先删除列表中的重复项并返回一个字典，最后将其转换为列表。此方法也可用于字符串，之后列表中元素的顺序也发生了变化。
# removing duplicated from list # using collections.OrderedDict.fromkeys()from collections import OrderedDict# initializing listtest_list = [1, 5, 3, 6, 3, 5, 6, 1]print ("The original list is : " + str(test_list))# using collections.OrderedDict.fromkeys()# to remove duplicated from list res = list(OrderedDict.fromkeys(test_list))# printing list after removal print ("The list after removing duplicates : " + str(res))
5. 处理嵌套列表中的重复元素
用于多维列表（列表嵌套）重复元素移除。这里假设列表（也是一个列表）中具有相同元素（但不一定是相同顺序）的元素被认为是重复的。
使用set() + sorted()方法：
# removing duplicate sublist # using set() + sorted()# initializing listtest_list = [[1, 0, -1], [-1, 0, 1], [-1, 0, 1],[1, 2, 3], [3, 4, 1]]# printing original listprint("The original list : " + str(test_list))# using set() + sorted()# removing duplicate sublistres = list(set(tuple(sorted(sub)) for sub in test_list))# print resultprint("The list after duplicate removal : " + str(res))# 输出结果：# 原始列表：[[1, 0, -1], [-1, 0, 1], [-1, 0, 1], [1, 2, 3], [3, 4, 1]]# 去重后的列表：[(-1, 0, 1), (1, 3, 4), (1, 2, 3)]
使用set() + map() + sorted()方法：
# removing duplicate sublist # using set() + map() + sorted()# initializing listtest_list = [[1, 0, -1], [-1, 0, 1], [-1, 0, 1],[1, 2, 3], [3, 4, 1]]# printing original listprint("The original list : " + str(test_list))# using set() + map() + sorted()# removing duplicate sublistres = list(set(map(lambda i: tuple(sorted(i)), test_list)))# print resultprint("The list after duplicate removal : " + str(res))# 输出结果：# 原始列表：[[1, 0, -1], [-1, 0, 1], [-1, 0, 1], [1, 2, 3], [3, 4, 1]]# 去重后的列表：[(-1, 0, 1), (1, 3, 4), (1, 2, 3)]
Reference：
